Lessons Learned and Quick Fixes

Saturday, January 28, 2017

How to create a .NET Core MVC File Uploader Module - Part 1

Fine UploaderEver since I started developing websites, file/image uploading really gave me a hard time because I just couldn't figure out how to make the upload more generic and pluggable. After seeing Google/Facebook/500px♥ does it, I figured, it must be possible and should be simple. I always tend to over-complicate stuff. A prime example is this SO question (Adjusting dates in an array based on a start date) I asked a few days back. After a few tries, I came up with a solution. This is done using .NET Core MVC6 and You are free to steal this and modify according to your needs. (Give credits where necessary, be human)

Uploader

So my requirements are simple:
  • To be able to upload files into a central file repository
  • To be able to select uploaded files and attach them to a post
  • If it's an image, show a preview
  • To be able to add metadata related to the file as we upload them
  • To be able to validate the said metadata before uploading
  • To be able to plug it into any page without too much configuration
  • To be able to extensively configure the module/plugin (whatever you wanna call it) 
  • To be able to use more than one module at the same page without conflicts
  • To be able to do the usual CRUD operation on the files
many instances

There are many file uploader scrips readily available on the internet but my above needs required me to make many of those plugins to work together. None of this would be possible if it's not for the infamous Head.js library. So, the plugins I've used are:
  1. Head.js - script/css loading
  2. Fine Uploader.js - File uploading without the hassle
  3. knockout.js - data binding
  4. underscore.js - lots of helpers
  5. Select2 - nice searchable dropdowns
  6. Bootstrap 3 - layout/css
Apart from these, my dev environment is MVC6, .NET Core, VS2015 & C#

So, lets begin:

Step 1: Planning

If we jump right into code when we have a task at hand, we have to get ready to do a lot of iterations to get things right. That, is a hard earned lesson. So, lets plan first. In its most simplest form, I need to be able to do something like @FileUploader.SetupHere(config) but I felt like its' just too much work for something simple. May be later I can put this into a nice little library and set it up that way. So until then, something like @await Html.PartialAsync("Widgets/FileUploader/FileManager", FileSelectorSettings.ImagesOnlySettings) would get the job done.

layout

I have a file manager module, which contains file select module and upload module. Right now, my file select UI looks like below. And file upload UI looks like the first image of this post. (Yeah!)

file selector ui

File select view will load its own scripts/css and knockout view model to handle file selection.
File upload view will load its own scripts and css. So we have a nice separation of concerns here. And head.js will load our files after the document is ready, therefore we have nothing to worry about having missing scripts.

Code:

First , lets start with the settings class. Code is pretty straightforward. I've defined most of the configurable aspects of the module. There are many more and this class can be cleaned up and organized neatly, which I will do at a later date.

I will show you the javascript on the next post since this one is too long already.

Thanks. and stay tuned.

Tuesday, September 13, 2016

Token Authentication C# Console Application - Sample

How to use token authentication from a console application to call and Web API endpoint.

Sunday, June 5, 2016

ASP.NET MVC Fine Grained Identity & Access Control - Part 3

In this series I will explain how I designed a generic claims based access control system ASP.NET MVC 5 with Identity 2.0.

In Part 1, I describe my approach and the initial database design. ASP.NET MVC Fine Grained Identity & Access Control - Part 1

In Part 2, I started laying foundation for the whole system with all the data models and a functional UI.

In this Part 3, I will start with most of the coding to make it work.

So far completed coding is in the GitHub repo: https://github.com/dirnthelord/ClaimsAuth

Note that, in part 2, I've mentioned the functionalities I want the system to have. In this post, I will complete all the core functionality so that I can directly edit the DB table values (using SSMS - SQL Server Management Studio) to test things out. I still need to create a UI for the super admins to easily manage permissions.

First off, lets grab all the Areas, Controllers and Actions in the system. These are what you going to lock down to certain groups.

I have posted the code for doing this a while back. This is it.

When you call 'ResourceHelper.GetResources()' it will give you a list of all the areas, controllers, actions for MVC/WebAPI in the system. Then you have to save these in the db. I choose to do this task in background in the Global.asax Application-Start method.

I will list down the tasks I need to do to make this work.
  1. Store all the Areas, Controllers and Actions in the DB
    • If the route exists, I don't need to insert it again. So to do this, first, I retrieve all the app-resources compare against the newly extracted, save only those which are new. This way, I don't have to manually enter anything every time a new feature gets added to the system. As long as I follow a clean Area-Controller-Action separation security is dynamic and adaptable.
  2. Retrieve the group permissions from the db
  3. Put them all in a dictionary for fast access.
  4. Once a user logs in, we check their group(s) and setup claims for each route. 
  5. For admins, we use special claims so we can reduce the number of claims
  6. For other users, we have to specify the claims they have.  
First, I'm gonna use my ResourceHelper class to get all area-controller-actions. and gonna save them in the db.

To do that, I have created a class called APM (Application Permission Manager) and have "SaveResources" method call 'ResourceHelper.GetResources()' and save them in db in a background task.

    /// <summary>
    /// Application Permission Manager. 
    /// Calling APM for short
    /// </summary>
    public class APM
    {
        //... rest of the code (check source)
        public static void SaveResources()
        {
            TaskHelper.RunBg(() =>
            {
                var resources = ResourceHelper.GetResources();
                Trace.WriteLine($"Saving {resources.Count} Resources");
                var repo = new ResourceRepository();
                repo.Save(resources);

                Initialize();
            });
        }

Following code shows my logic to save only the new ones, along with any newly created anonymous resources.

public class ResourceRepository
{
 //...other code (check source)

 public int Save(List<ApplicationResource> resources)
 {
  // get all existing/saved resources
  var existing = GetResources().ToList();
  // select only the ones which are new
  var itemsToInsert = resources.Where(item => !existing.Contains(item)).ToList();

  var result = 0;

  // go through each new resource and save them, one by one(for now.)
  foreach (var item in itemsToInsert)
  {
   try
   {
    SaveResource(item);
    result++;
   }
   catch (Exception ex)
   {
    Trace.TraceError("Unable to save resource: " + item.ToPermissionValue(), ex);
   }
  }

  // get all existing/saved global permissions
  var existingGlobal = GetResourceGlobalPermissions().ToList();
  // find the resources which can be accessed anonymously
  var anon = resources.Where(x => x.IsAnonymous).ToList();
  // get all the newly saved anonymous resources
  // because we have to add a record of them to global permissions table.
  var savedAnon = GetResources().Where(anon.Contains).ToList();

  foreach (var anonItem in savedAnon)
  {
   try
   {
    // save only the new ones. new ones doesn't exist in the existing globals list. (yet)
    if (existingGlobal.All(g => g.ResourceID != anonItem.ID))
    {
     SaveResourceGlobalPermission(new ResourceGlobalPermission(anonItem.ID));
     result++;
    }
   }
   catch (Exception ex)
   {
    Trace.TraceError("Unable to save resource global permission: " + anonItem, ex);
   }
  }

  return result;
 }

 //... more code (check source)

}

Now that we have all the resources saved, what's left is retrieve them, cache them and use them to make sure no one gets access to anything. (Everything should be locked by default)

I have updated the connection string to point to a SQL Server database. Should auto-generate the db when you first run the application, and save all the available resources. I count 56 total resources.

I'll update the code and write another post on the next steps soon...

If you have any questions, feel free to drop me a message, anywhere you like.

Sunday, April 17, 2016

ASP.NET MVC Fine Grained Identity & Access Control - Part 2

In this series I will explain how I designed a generic claims based access control system ASP.NET MVC 5 with Identity 2.0.
In Part 1, I describe my approach and the initial database design. ASP.NET MVC Fine Grained Identity & Access Control - Part 1

In this Part 2, I will start laying foundation for the whole system with all the data models and a functional UI.



So far completed coding is in the GitHub repo: https://github.com/dirnthelord/ClaimsAuth

First, I need to decide what functionality the UI should have.
  1. As a super admin, I need to be able to view the application resources from the UI. Add a description and search them.
  2. As a super admin, I need to be able to manage all the users in the system.
  3. As a super admin, I need to be able to manage global permissions
  4. As a super admin, I need to be able to manage user groups and group permissions
  5. As a super admin, I need to be able to manage user groups and group permissions

So, these are the list of things I need to be able to do within the system. There can be more, depending on the requirements. But I’m just trying to present the most basic way of doing things.

I also need a way to test things out within the UI. To do that, I need to be able to login with another less privileged user and see if everything I’ve blocked, stays blocked. Typing the URI manually, must not let the user view what’s hidden.

A tree structure for the application resources will display them nicely. If I could select the user group first, and then load the already given permissions and map them into the tree with little check-boxes on it, will give me an easier way to manage permissions for the group. (see the following image from WSO2 API Manager Permission Model)

WSO2 API Manager Permission Model

I can pre-define the types of exclusive rights in the system so that I can eliminate human error. Admins can select the right/claim and assign values to them.

These are my design considerations for now. As I keep developing the system, my decisions might change.

I’m going to start with a sample MVC-5 + WebAPI enabled project with Individual user accounts.

MVC + WebAPI + Individual User Accounts
I have removed all the bootrap and JQuery JS and CSS from the sample project and linked CDN supplied libraries instead. I just wanted to keep things clean.

I have created new MVC controllers and the models from my DB design.


Note that, I didn't run the application yet. If you run it, and register, the database will be created. I just like to setup the db with everything I want first.

Connecting the data models to my ApplicationDBContext.

using System;
using System.Data.Entity;
using System.Net.Mime;
using Microsoft.Ajax.Utilities;
using Microsoft.AspNet.Identity.EntityFramework;

namespace DinkLabs.ClaimsAuth.Web.Models
{
    public class ApplicationDbContext : IdentityDbContext<ApplicationUser>
    {
        public IDbSet<Application> Application { get; set; }
        public IDbSet<ApplicationResource> ApplicationResource { get; set; }
        public IDbSet<ResourceGlobalPermission> ResourceGlobalPermission { get; set; }
        public IDbSet<ResourceRolePermission> ResourceRolePermission { get; set; }
    
        public ApplicationDbContext()
            : base("DefaultConnection", throwIfV1Schema: false)
        {
        }

        public static ApplicationDbContext Create()
        {
            return new ApplicationDbContext();
        }
    }
}

The final UI looks like this.


A few menu items are still missing, I'll be adding them on  Part 3.

Sunday, April 10, 2016

Why You Should Always Use Decorator Pattern

Life get easier when you have more reusable decorators and a good dependency injector.
image credits: http://www.homedit.com/how-tell-difference-interior-designer-interior-decorator/


Years before I choose this programming/engineering career, I always wondered, how can I make something better? How can I make it work faster or efficiently? I'm no backyard scientist or whatever so basically, it was all just thinking and speculating. Running simulations in my head, kinda. Once I started programming, I was interested in making machines talk. "Knight Rider" was my inspiration. Talking cars!!! And then, I was super interested in learning all the mechanisms behind AI. I'm a fan of AI, taking over the world or not, I always wanted to build one. For some fucked up reason, I still can't even get a simple neural-net working. Sure, copy pasting works but what's the point. I suck at math, may be that's the reason. And, as with everything in life, we come up with excuses to either postpone something or completely give up and look the other way. I just never, completely gave up, I just postponed, until I find or read about a better way. As with everything else in nature, software is always evolving. Programming languages evolve, to make our life easier. Hardware evolves to make shit happen way faster than it was a few months ago.  AI tech evolves, to make machines think better.

I had read about the pattern of decorator. To decorate is to cover something ugly with something pretty or simple. I don't wanna go into all the technical definitions as to what the decorator pattern is. but, I'll just copy it from somewhere (Wikipedia).
In object-oriented programming, the decorator pattern (also known as Wrapper, an alternative naming shared with the Adapter pattern) is a design pattern that allows behavior to be added to an individual object, either statically or dynamically, without affecting the behavior of other objects from the same class.[1] The decorator pattern is often useful for adhering to the Single Responsibility Principle, as it allows functionality to be divided between classes with unique areas of concern.[2]
It says something about Single Responsibility Principle, one of the SOLID principles in OOP. I saw the term SOLID a few years back and then I didn't understand shit. I thought, if it works, why the fuck would I wanna break it and put all the patterns and shit into my code, and make it so complex. Well, I was obviously stupid. As with all things, I did get past some of that stupidity, evolved into someone who appreciates the better ways of doing things.

Months ago, I have put the code to measure execution time, and parameter/result logging and bunch of other cross-cutting (logging, monitoring, caching, validation etc.) Stuff in the same method, and I was a fan of God classes, even though I felt something is not right, I just went with it. Time constraints and chats and just too lazy. I had replicated/copy-pasted the same code, on many places. and when I realize I am copying the same code, I would extract a method, with an 'Func' or 'Action' type parameter, and then I would put all the god logic in that method. Something stupid like this.

public int SaveContact(Contact contact)
{
 var st = new Stopwatch();
 Log.InfoFormat("saving customer params: {0}", contact.Json());


 if (contact.CustomerID == 0 || contact.UserID == 0)
  throw new ArgumentException("Customer ID or User ID is missing");
 st.Start();
 try
 {
  using (var repo = DataServices.DataRepository<SMSContact>(UnitType.Contacts))
  {
   var saved = contact.ID > 0
      ? repo.Update(contact)
      : repo.Add(contact);

   return saved.ID;
  }
 }
 finally
 {
  st.Stop();
  Log.InfoFormat("saving customer took: {0}ms", st.ElapsedMilliseconds);
 }
}

You might think, what's so stupid about this? Well, nothing is stupid, until you have so many method, doing the same logging/measuring/caching etc. behavior. It's just too much copying going around. Imagine, someday, you come across this super cool open source execution time measurement library and you decide you want to put it in all your methods, OR imagine someday you realize that, there's too much logging going on OR you don't really need the measurements and logging at all, OR imagine someday you decide you want an extra caching layer between your API and business rules or whatever. How many methods will you have to modify. Trust me, it's a very boring task. Been there, done that. Obviously, these apply to you only if you always wanna make yourself better. If you are the kinda person who would just hack it and get it working and not look at it again, then certainly, you are making your life and the one after you very miserable.

How would I get around this endless replicating code? An abstraction like the following.

protected static T Execute<T>(Func<T> codetoExecute)
{
 var st = new Stopwatch();
 st.Start();
 
 Log.Info(m => m("[<{0}>] Scope Begin >>>>>>>", typeof(T).Name));
 
 T result;
 
 try
 {
  Log.Trace( "Executing method: " + codetoExecute.Method.Name);
  result = codetoExecute.Invoke();
 }
 catch (AuthorizationValidationException ex)
 {
  Log.Trace("<<<<<<< Scope Error : AuthorizationValidationException >>>>>>> " + ex.Message);
  throw new FaultException<AuthorizationValidationException>(ex, ex.Message);
 }
 catch (FaultException ex)
 {
  Log.Trace("<<<<<<< Scope Error : FaultException >>>>>>> " + ex.Message);
  throw;
 }
 catch (Exception ex)
 {
  Log.Error("<<<<<<< Scope Error : Exception >>>>>>> " + ex.Message, ex);
  throw new FaultException(ex.Message);
 }
 
 st.Stop();
 Log.Info(m => m("<<<<<<< [<{0}>] Scope End, time took: {0}ms", typeof(T).Name, st.ElapsedMilliseconds));
 
 return result;
}

Now, we can call do our stuff like as follows.

public int SaveContact(Contact contact)
{
     return Execute(() =>
     {
          // do stuff
     });
}

Problems solved, no more copy pasting. I was fine with this way of coding until I came across this blog post, by the author of "Simple Injector". One day I wanted a better way of handling the hotel booking flow I was working on and I thought may be Command Query pattern or something will do the magic. I went on a quest to find out what I can do better. The great oracle, the Google.com pointed me towards this blog post. Meanwhile... on the command side of my architecture .

This article describes how a single interface can transform the design of your application to be much cleaner, and more flexible than you ever thought possible.
This headline got me hooked, I read on. And then I read all his blog posts. This was exactly what I was looking for, to a different problem. Nothing pointed me towards a better search mechanism for the problem I had. Who cares, it already works. This was something new and I wanted to try it out. Decorators!!

However, the command and query patterns described in the blog didn't quite fill my needs. Because, commands usually alter state. It doesn't return anything, while queries, doesn't change anything, it just returns whatever the shit you're looking for. I wanted a way to alter state and return shit back to me. Thus, instead of "void" why not return "T"?

There might be better ways of doing the command and query work. One possibility I can think of is do the command in a background task and query when the task is complete. I don't see the need to go into such lengths yet! maybe my POV might change, in the near future, thanks to another brilliant person.

I understand no one seem to share these code online. I had to do a lot of research to get to this point. That's what experience is all about, if everything is given to you on a silver plate, you will be just as dumb as a monkey. You need to do the worst and do your research and evolve your skills into something better.

Damn, this might be the longest blog post I've created. Sunday evenings!

So, this is how I do the decorations of my travel booking engine, especially the command side.

As you can see, the 'CreateBookingCommandHandler' is my command handler implementation. I haven't put any logging code there. nor error handling. It just deals with calling the validation provider (I use FluentValidation btw) and the 'IBookingPreProcessor' handles all the shit I do after a successful validation. And then I save the booking and return the results.

Just like the 'MeasuringCommandHandlerDecorator' I can create whatever decorator I want to use, and plug them in my injector configuration, as follows. I can turn on or off (Comment/Un-comment) any decorator I want, anytime I want and it won't even affect the system, despite the slight performance gains.

This is how I come to love having decorators throughout the system. Reach out to me if you have any questions regarding these or for any corrections I should do. Once again, as with all things, If I come across a better way of doing these, (apart from AOP), I'll update the post.

Friday, March 25, 2016

Custom Error Messages in ASP.NET MVC

Even when you have custom errors mode on,
and Have all the custom error pages setup, like the following..

    <customErrors mode="On" defaultRedirect="~/Error" redirectMode="ResponseRewrite">
      <error statusCode="403" redirect="~/403" />
      <error statusCode="401" redirect="~/401" />
      <error statusCode="404" redirect="~/404" />
      <error statusCode="409" redirect="~/409" />
      <error statusCode="500" redirect="~/500" />
    </customErrors>


You even set Response.TrySkipIisCustomErrors = true; on your error handler to get rid of those ugly IIS default error pages.
      
        /// <summary>
        ///     Returns a Not Found error.
        /// </summary> 
        [HttpGet, Route("404")]
        [AllowAnonymous]
        public ActionResult NotFound()
        {
            Log.Error("Not Found - 404 : {0}".FormatWith(HttpContext.Request.RawUrl));
            HttpContext.Response.Clear();
            HttpContext.Response.ClearHeaders();
            HttpContext.Response.StatusCode = (int)HttpStatusCode.NotFound;
            HttpContext.Response.TrySkipIisCustomErrors = true;

            return View();
        }

But still, nothing works. Until you set the following config line.


<system.webServer>
    <httpErrors existingResponse="PassThrough"/>
</system.webServer> 

Wednesday, January 27, 2016

Beautiful Code Makes Life Beautiful

This is the Part 1 of my post series about good code.

Part 2 : Why I Always Choose Decorator Pattern

There are two kinds of software developers. Those who write crappy code and those who write beautiful code. I'm sure if you are a fan of beautiful code, you'd agree with me. Those who don't agree and those who think I'm stupid, read on.

Image credits: http://xpconference.in/
Everyone has to start somewhere, right? Us, developers, when we start creating our first program, most of us don't really know what design patterns are or don't understand them or why we should be using them. We just jump right into writing code. And then we hit run and the code works. So fucking great!

Atleast, that's what I thought before I start making a living out of my code. Organizations need readable, maintainable, bug free, well commented and well understood code. Imagine we were tasked with adding a new functionality to assembly code. I don't even know assembly because that shit is ugly. Sure, we can talk to the machines at the lowest level, but why bother? This isn't 1960's anymore.

I've read somewhere that software engineers strive for well crafted maintainable code while developers just want to get something done faster. Developers don't really have to care about how maintainable their code is. Engineers does. Because, If you build a bridge without a proper plan, bride is doomed to collapse.

If you used to be a developer ages ago and now you are a engineer or an architect, then have you ever came across your ages old code and had this reaction?

What the fuck is this shit? I can't believe I wrote this shit. yuk! 
I did, my friends did. Probably a million other developers did too.

Take a look at this code. Those who think this is beautiful and this code will make your life beautiful. Thanks for visiting my blog. You don't have to read on. Instead, check out my personal site.

(Following code is missing a lot of classes and methods. This was something I wrote years ago to search hotels and filter them based on user selected criteria -facets- )

public class HotleSearchService
    {
        private Log _log = new Log();
        public HotelResult Search(HotelQuery query)
        {
            query.Filters.Add(new Filter
            {
                Name = "Price",
                Expression = res => res.Price.Between(query.SelectedFilters.Price)
            });
            query.Filters.Add(new Filter
            {
                Name = "Check In Date",
                Expression = res => res.CheckInDate.Between(query.SelectedFilters.Date)
            });
            query.Filters.Add(new Filter
            {
                Name = "Check Out Date",
                Expression = res => res.CheckOutDate.Between(query.SelectedFilters.Date)
            });
            query.Filters.Add(new Filter
            {
                Name = "Facilities",
                Expression = res => res.Facilities.Contains(query.SelectedFilters.Facilities)
            });
            query.Filters.Add(new Filter
            {
                Name = "Property Types",
                Expression = res => res.PropertyTypes.Contains(query.SelectedFilters.PropertyTypes)
            }); 
            query.Filters.Add(new Filter
            {
                Name = "Rating",
                Expression = res => res.Rating.Between(query.SelectedFilters.Rating)
            });
      

            _log.Info("New hotel search request" + query.ToJson());
            var st = new Stopwatch();
            st.Start();
            var service = new HotelClient();
            var results = service.Search(ConvertRequest(query)); 
            st.Stop();
            _log.Info("hotel search took : " + st.ElapsedMilliseconds +  "ms");
            _log.Info("applying filters");
            st.Start();

            foreach (var filter in query.Filters)
            {
                results = filter.Apply(results);
            }

            st.Stop();
            _log.Info("hotel search took : " + st.ElapsedMilliseconds + "ms");

            var x = Convert(results);
            x.DoCalculations();
            x.LoadCached();
            return x;
        }
    }

Above code is messy. Does too much in one class. Which clearly violates Single Responsibility principle of OOP. I have called it search service, so it should only do searching. But, it also defines filters, do logging, measure execution time, filters results, converts results. The complete class will be thousands of lines of code. Maintaining this shit is an obvious nightmare. I know what it does, I know where to change. but what if a new developer comes and she was tasked with adding a new filter or add a completely new functionality like caching frequent searches and relevant results? We are looking at a lot of sleepless nights and depression here. Of course, I can do it in a day. But maintainable software should be maintainable by others too. Not by the one who originally wrote it.

From my next post, I will discuss how to make this code more readable and maintainable. I will follow SOLID principles, best practices and patterns as best as I can and will try to make this code beautiful. Making our lives beautiful along with it..

Part 2 : Why I Always Choose Decorator Pattern