Wednesday, 30 December 2015

Introduction to the Internet of Things: What, Why and How

Content

     

 1. Background

There are several articles, infographic and tutorials that defines and explains internet of things. But then the question is why to write another article on the subject. As a developer when I look around internet to find more information about IoT, I get plenty of "conceptual" articles of what IoT is but very few elaborates the details from a developer perspective. Also it does not get immidiately clear as to what kind of hardware and devices, software and services falls under IoT. Also what basic and migration skills a developer needs in IoT are also not clear from the basic articles. So in this article, I would focus to elaborate IoT as a developer perspective and would try to narrow down the more generic topic into specificities. I would also try  to elaborate upon the basic tools you require to start with IoT development. So if you are a developer looking to know about IoT, this might just be the right article for you to bookmark. But if you are a marketing guy wanting some information for your presentation, save your time.

2. What is Internet of Things (IoT)?

2.1 Embedded System

Wikipedia says 
Quote:
The Internet of Things (IoT) is the interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure.
So Internet of Things or IoT basically is connecting Embedded System to internet.  So before we understand IoT, let us understand Embedded System a little.
 The heart of the embedded system is a RISC family microcontroller like PIC 16F84/Atmel 8051/ Motorola 68HC11 and so on. Most important thing that differentiate these microcontrollers with microprocessors like 8085 is their internal read/writable memory(EPROM). So You can develoop your light weight program ( in Assembly language or using Embedded C with software like Keil) and "burn" the program into the hardware. These programs keep on running in a loop. 
Interestingly in most embedded system a single program is burn with several subroutines. So unlike your PC, microcontroller device in an embedded system runs a single program infinitely.
You can connect several input and output devices with these microcontrollers which are either memory mapped or IO mapped. These simple hardware includes LCD display, buzzers, keypad (numpad) or even a printer. You connect several sensors through A/D interface. The devices can control Higher Power/Voltage/Current rating devices like fans, motors, bulbs using drives devices like relay-optocoupler and so on.
Figure 2.2 Shows a typical Coke vending machine and it's internal embedded system.
You can clearly see the switches, LCD and relays in the figure on the right.
Form simple DIY robots to more commercial systems like washing machine, ovens, AC controller, Cars,Hand held ticket printers, Hotel Mini Bill printers embedded system is everywhere around us. 
So why  such embedded systems are attractive?
1) Autonomous: That means you can build a system specific to a particular application. For instance some standard peripherals and a specialized program can turn a microcontroller unit into washing machine controller or an oven controller. Also embedded systems can be built specific to the requirement. Unlike a PC which you can not work with without a monitor, an embedded system may not mandatorily need a display unit.
2) Low Cost: The cost of the microcontroller unit is magnitude scale lower than a full fledged computer.
3) Low Space: Obviously figure 2.2 shows that an embedded system takes way too little space when you compare it with even the sleekest of laptops. So you can put such a system in your hardware system quite efficiently.
4) Low power: Most of the common microcontrollers out there which are popular choices for embedded systems operate at 5V and often need 5v regulated power supply which can be provided through a simple 9v standard battery with voltage regulator or directly from main by using a voltage rectifier with filter circuit. 
There are many other factors which includes I/O speed and cost, energy consumption par instruction and so on. So we get a fare deal of understanding of what an embedded system is and what contructs an embedded system.
But when you look at the miniature embedded system on right of figure 2.2, you see no keyboard. So how such microcontrollers are programmed? This is generally done by interfacing the microcontroller with PC. The kit is commonly known as "Programmer Kit" or "Burner". A typical atmel kit is shown in figure 2.3
Figure 2.3: Programmer kit for Atmel Microcontrollers
From device to device such a kit or the hardware required for burning a program in the microcontroller varies, but their general structure remains same. So the microcontroller of the system is put in the slot and interfaced with PC. Then a MC burning tool like Keil . We can use an Assembly program or Embedded C to write and burn program in the microcontroller. Finally we can connect and solder the other components in the final pcb to get the system working.
Well, that's how a general embedded system worked and probably will continue to do so.  
The important takeaway from the above discussion is that Microcontrollers in an embedded system needs PC interfacing for buring the program. But can the program itself communicate with PCs?
Most of the Modern microcontrollers are capable of Serial Communication. That is they can have a half duplex communication with PC using Serial port over either RS-232 or with USB port if a USB driver circuit is added.The advantage of such a connection is that data from the microcontroller can be logged also microcontroller can now communicate with more advanced programs like Java or C#.Net as most of these advanced programming languages supports serial communication.
For example suppose you are building a simple temperature display embedded system with fire alarm. A LM-35 sensor is connected with the microcontroller through 8 bit ADC. Microcontroller is connected with a 16-pin dual line LCD as seen in figure 2.2. So it keeps measuring temperature and displays in the LCD. Once the temperature exceeeds certain threshold, it triggers an alarm.
With the help of PC interfacing the temperature can now be displayed in PC monitor and periodically can be logged into a file or database for the the values to be analyzed at a later stage.
 So even certain basic operation like getting a bare embedded system to work needs extra hardware as the core microcontroller is just a single chip. The MC needs power circuit, ADC circuit, interfacing circuit and so on.
Therefore standard kits were developed which has this standard set of hardware embedded into a single board which can be used readily without having any external hardware.

2.2 Embedded Boards 

In order to standardize the hardware and associated software and provide basic and bare minimum hardware units required to have an embedded system work, standard embedded boards are developed with on board hardware components like voltage regulator, USB FTDI chip, A/D converter and so on. Different boards supports different Micrcontrollers and different CPU architecture. These boards are tailored made for particular applications like medical electronics, gaming and so on. Have a look at the image below for some sample boards.
Figure 2.4 Some Common Embedded Boards ( source :www.eurotech.com)
IDE ( integrated development environment) were developed for easy burning, debugging and developing software and firmware for production level system. So instead of dealing with single microcontroller unit and debugging and maintaining hardware in the chip level, you can now have more compact kits ready for more complicated needs.
However the kit that truely revolutionized Embedded Kits and given a wing to DIY guys was Arduino.  Though in this Article we will not try to teach anything about Arduino, we will acquire some basic understanding of Arduino which will help us understand IoT embedded boards at large.

2.3 General Introduction to Arduino Boards

Quote:
Arduino is an open-source electronics platform based on easy-to-use hardware and software.
Arduino is an architecture that combines Atmel microcontroller family with standard hardware into a board with inbuilt bootloader for plug and play embedded programming. Arduino Software  comes with an IDE that helps writing, debugging and burning program into Arduino. The IDE also comes with a Serial Communication window through which you can easily get the serial data from the board.

Figure 2.5 A Typical Arduino Board (Dicimilia)
This board has everything starting from ADC to USB controller to PWM ports. So you can now control relays or connect LCD into the digital ports on the top and directly connect the output pin of the sensors to analog ports in the bottom. 5v or 3v power required to drive different sensors can be directly taken from the power port on the bottom left side. The board can be powered either by USB or by connecting 9v battery in pin Vin. You can also drive the board with a standard 9v adapter. The pins are such that you can actually connect several hardware without any soldering just by inserting the wire or inserting sensor/device pins into Arduino pin holes.
Arduino can communicate with PC using USB cable and it is configured as slave device. Multiple Arduinos can be interfaced togather by making them connect through serial communication ( see pin 0 and pin 1 on top right. They enable Arduino to communicate serially with other device). Arduino can also become master to other peripherals like Fingerprint sensors which can be connected to Arduino through Tx-Rx pins.  
 I have developed several hobby and prototype projects using Arduino. You could sometime checkout myYoutube Channel's Arduino playlist 
Due to it's open hardware and software architecture you can really build your own board with your own requirement. However different boards of different size, architecture, power requirement and utility are already available in the market. Take a look of wide range of Arduino boards in figure 2.6.
Figure 2.6 Different Arduino Boards
We are not really trying to learn Arduino in this tutorial, but rather our aim is to understand IoT with the help of Arduino.
So with this knowledge that basic embedded system can be provided as a board with certain plug and play capabilities, we are more or less  ready to understan IoT has a whole.

2.4 Formal Introduction to IoT

Recall that we defined IoT as embedded devices which could be connected with internet. Having understood what an embedded system is and what a modern embedded board is all about it is not too difficult to perceive the idea of IoT.
Consider the following schematic of a temperature sensor being connected to Arduino
Figure 2.7 Schematic of Temperature sensor LM-35 Connection with Arduino
So using Arduino's IDE we can establish a Serial communication with the device and can read temperature value. Right? Or we can interface a LCD display to display the temperature. But how about checking out the temperature from any part of the world using internet? 
How about getting the temperature information in your mobile phone? or Getting the temperature as a tweet after certain interval?
Doesn't the scheme looks attractive? Yes it surely does. If you can connect your embedded device into internet, you can get sensor information online which can be viewed in wide range of devices including your tablet and mobiles. You can also control devices over internet. You can actually have several home appliences connected to your embedded system and the embedded system being connected to internet with unique ip address. Then you can actually instruct the device to yurin on or off certain peripheral devices by generating the instruction online. 
So Internet of Things or IoT is an architecture that comprises specialized hardware boards, Software systems, web APIs, protocols which togather creates a seamless environment which allows smart embedded devices to be connected to internet such that sensory data can be accessed and control system can be triggered over internet.
Also devices could be connected to internet using various means like WiFi, Ethernet and so on. Furthermore devices may not needed to be connected to internet independently. Rather a cluster of devices could be created ( for example a sensor network) and the base station or the clusterhead could be connected to internet. This leads to more abstract architecture for communication protocols which ranges from high level to low level.
Following diagram explain what is IoT all about.
Most interestingly, these devices must be uniquely discovered. For unique discovery of the devices in a Network, they need to have unique IP address. As number of IoT devices online is expected to supress 20 billion mark and that IPv4 can only support upto 4Billion unique addresses, IoT devices essentially have IPv6 addressing scheme. All these devices have either fixed or Subnet masked IP addresses of type v6.
Unique IP addresses makes IoT devices discoverable in the internet as independent node. This is the most important concept to have in mind to understand IoT.

 

3. What Devices Make it to IoT

Before we see what devices make it to IoT, we need to clarify a few more things here. So far we have seen that IoT are essentially emedded systems and smart objects connected to internet with unique IP address which can be discovered and communicated over internet. We have also seen that the IoT devices may have external peripheral like Actuators and Sensors.

3.1 Are Mobile Phones IoT devices?

One of the most common we see in day to day life are mobile phones. Mobile phone is essentially an embedded system with a processor at the core having display and keypad. They support wide variety of sensors like Ambient light Sensors, Accelerometer, Gyroscope and so on. They are connected to internet. Mobile phones gets IP addresses, can access internet. In other words it virtually fits every description of IoT. So can we call mobile phones IoT devices?
This doubt was clarified at a keynote event during Sept 2011's Mobile World Congress in Barcelona by Qualcomm Chairman and CEO Dr. Paul Jacobs.
Paul Jacobs talked about how mobile technology could be used to connect non-phone, non-tablet devices called IoT devices and objects to the Internet. In this future where everything is Web-connected, mobile phones will serve as the hub, or the remote control, for Internet of Things.
So IoT is internet connectivity of smart objects and embedded system other than mobile phones which can be connected with external hardware and Mobiles, Tablets, Laptops and PCs are remote control/access center of IoT. 

3.2 What are Smart Objects?

We find the mention of "Smart Objects" and "Smart Devices" quite often in general defination and description of IoT alongside network connectivity of embedded system. But what exactly are smart object?
Quote:
A Smart Object is an object that can describe its own possible interactions. 
  Any object which not only has a state, which has certain data associated with a state but an object which can also determine nature of connectivity, duration of connectivity and connectivity protocol are called smart objects.
Radio Frequency Identification (RFID), Bluetooth Low Energy (BLE), and Near Field Communication (NFC)  makes it possible to use our phone as readers. We can extract information from certain objects just by tapping it or bringing our device close to it. RFID tags does not have any embedded system nor does a NFC tag has. But data can still be brought to internet by reading through a reader. These are called smart objects.  Non-processor entity whose data can be acquired and migrated over internet falls under these category. 
Short-range low-energy sensor technology has given rise to smart objects. So say a T-Shirt comes with a smart tag. We can tap our phone over it and we get information about the quality of material, it's size, other color varients, dye information and so on which helps us making our purchase decision. As the information is acquired, it can be passed as a search query to obtain similar results where we can compare the price-quality for similar tags.

3.3 IoT Devices

If you keep track of the Sillicon Vally news then you are sure to know that IoT is the latest buzz. So many of the startups now are focussing towards IoT. Therefore when we are talking about IoT, we are basically talking about infinite possibilities, hardware and software platform. So when you try to find a list of IoT devices in internet, you might end up getting frustrated purely because it's is difficult to provide such list.
Therefore research of this particular subsection has taken significant time. Finally I have put togather somemost common and popular technologies in IoT to give you an overview of what devices are we really talking about.
We divide the IoT devices into two braod categories: The wearable ones and Microcontroller/Microprocessor driven embedded IoT devices. Some of the Embedded devices like Arduino Lillypad are minisque and you can further utilize them to make your own wearable solution. But in wearable I have included hardware which are pretty standard and IoT has only software scope for the developer.
I have also put some list oof common peripheral hardware that you might have to learn while working with IoT hardware in embedded level.
Figure 3.1 Common IoT devices and Technologies
So if you are not a hardware geek you can start with Wearable and start making apps for popular wearable platforms. If you are a hardware enthusiast, you can start with Embedded IoT platform. You may select any of the broader technologies like Raspberry Pi, Arduino or Galileo and startup with IoT development.
I am personally an Arduino fanboy and have done huge number of hobby as well as commercial projects in Arduino. I have also worked with PI. I can say that if your application is more "core hardware centric" ( means using actuators and sensors) Arduino is a great bet, but if your application is more data centric ( like creating a file server or a media server) then Raspberry Pi beats all other hands down. Galileo uses the same IDE as that of Arduino. So if you can learn Arduino, Galileo is an easy migration.  I am not ashamed to say that I have very much zero experience about the other hardware platforms.

4. IoT Platforms

At this stage we divide our IoT development into two parallel technologies: Wearable and Embedded. Developers can build apps for custom wearable devices like Peeble, Samsung Gear or can opt to create their own platform using Embedded solution and then can develop app for that platform.

4.1 Wearable Pltform

Tizen is fast becoming one of the most popular platform for Mobile and wearable devices. Tizen SDK comes ported with wearable emulator which makes it easier to develop wearable solutions for Tizen platform.
As fifure 3.1 suggests, a large Android Wear devices are now being made and marketed. Smart watches are getting popular by every day. Android Wear apps can be developed and tested in Eclipse. This Android Developer Guide helps you in setting up Android Wear development environment in Eclipse.
Salesforce is another platform which is coming up with awesome development environment, APIs in wearable technologies. Their solution is extended from Peeble to Google glass. Salesforce is really worth a try if you are planning to have a serious go at wearable technology as a career option. Checkout Salesforce Wear page.

4.2 Embedded Pltforms

Arduino  is probably the best starting point for embedded based IoT. Basic Android boards don't come with Ethernet shield and for Arduino to be able to work as IoT device, you need to select Android with Ethernet shield. Android Yun on the other hand is a board that comes ported with ethernet shield. You can actually order a basic board of Arduino like Arduino Decimilia or Dueminolova and learn the hardware basics like connecting sensors, working with actuators, serial communication and then you can go for Ethernet shield and look for more web based application for Arduino.
Raspberry Pi is probably one of the best things to happen in DIY IoT.  A wide range of Data driven applications like Home Automation Server toHome Multimedia server, File Server can be developed withPi. PI like Arduino has general purpose IO pins. But seamless working with sensors is bit tedious in Pi.
Another efficient IoT board is Intel Edision which has integrated BLE, WiFi among host of other features. It supports wide range of Industry standard hardware ( over 30) through 70-pin interface. What is important is it supports wide range of platforms including Arduino and Node.js
Intel Galileo is another good offering by Intel which supports the same shielding that of Arduino Uno. So it can be said to be first Intel powered divice which is Arduino compatible. It has among other thing a USB host controller like Raspberry Pi which makes this an attractive hardware. Galelio also has ethernet shield in built.
There was a time when Microsoft used to dictate echnologies and trends. Industries used to follow. That is no more the story. With several companies gunning for a space in wearable sector, microsoft seems to be doing all the catching up and does not look too impressive at this moment. None the less Netduino is a .Net Micro Framework based plstform where hardware is simillar to Arduino. But Netduino has 12 bit ADC as against 10 bit Arduino ADC channels and uses 32 bit Controller. There are few more differences. But the reason why Arduino is a better bet for me than Netduino is that I get an Arduino Dueminolova for under $10 where as Netduino is about $60. Though Netduino really has better multitasking, cost is a big factor for DIY guy.

4.3 Cloud Platform for IoT

Let's rediscuss the possibilities to beverage vending maching once more in terms of sheer possibilities. In the conventional vending machine you need to press a button or put a coin to trigger the process of liquid flow, which stops after certain quantity. Now how about integrating paypal or Google money with the vending machine? How about a customer discovering the vending machine as "website" along with it's location and then pays online for a glass ofbeverage. Once payment is successfull he gets an access token. He can pass the token to the machine through NFC and bingo he gets his drink.
Now this logical possibility is very important for understanding IoT and IoT really can bring several services ( like online payment gateway), several hardware platform ( like embedded board of the vending machine) and smart objects and data like NFC, GPS into a seamless environment.
Now if you can integrate online payment into beverage vending machine, why not in for a community washing machine? If you are using location service for beverage machine, then whey not utilize the location and payment service for the toll gate? Why not get the data of a medical diagnesis like ECG ( acquired through another embedded board partaining to medical electronics) into cloud such that several doctors can view it and form a comprehenssive opinion about the patient's state?
Well, infact all of them are possible. A little understanding of web and software design would take your mind towards cloud. Just like Web of Machines, in a Machine to Macine ( M2M) or Machine to Objects ( M2O) or any simimlar communication several modules will be common and several modules demands data to be available for sharing. Cloud APIs comes in handy in this regrad.
For instance when you have to make a device discoverable in web, you have to assign a fixed IP address, maintain a router and follow several networking skills. You might not have the knowledge and infrastructure needed for maintaining a commercial sophisticated network for IoT.
Yaler is a great example of what services and cloud can bring to table. This provides connection as a service such that your device is easily discoverable and communicable over the web without much hassle and take care of underneath security.
Axeda Provides infrastructure for M2M architecture.
OpenIoT is an open source IoT platform that provides out of other services a unique Sensing as  a Service. 
Google has already integrated location services with it's cloud. Location extracted from your devices are silently put in your status updates in facebook and twitter and are also used for more personalized searches.
So cloud APIs has a great potential in IoT in all levels of architecture starting from firmware to hardware to more top level architecture.

5. What IoT means for the developers

There have been few paradigm shift in computer industry. From Text operating system to Windows 3.1, then Windows 95, then internet revolution, the Google search engine revolution, eCommerce, Mobile phone, iPhone, Android. Each of these technological changes has seen many new opportinities being opened up for those who is eager to adopt to the changes. IoT is one such paradigm shift knocking our doors. It is opening up host of opportunities at both hardware as well as software level and would open up more standards in coming years. Hardwares will boil down to handfull of good baords, wearable will include few killing devices and platforms will have it's own popular choice like Visual Studio.Net and Eclipse of current computing platforms.
So, IoT really means that there is no stangancy and some cool opportunities calling the smart minds who can see it coming and can be ready by the time IoT actually takes the next giant leap.
at No comments:
Labels:

Thursday, 9 April 2015

Software Architecture - Examples

In this post I present how the same application can be created if it is done without any architecture, with an architecture completely bound to a technology and its limitations and with an architecture that I consider to be "unbound" from technologies and their limitations.
On the article Software Architecture I said that I usually start the architecture of applications by thinking only on how to solve a problem using the idea of a technology, not considering any limitations of actual implementations. Only later I try to chose a technology and in many cases decide to write my own because the existing ones don't accomodate my needs.
I know it is hard to get that idea without examples, so in this post I will try to show application designs ranging from the complete lack of architecture to an architecture bound to a technology and also an architecture that I consider unbound to technologies and their limitations.

Fictional Purpose

Create a simple web application that lists categories (with any level of sub-categories) and products. Such an application will not edit anything and there's no e-commerce at all. The data will be entered using another application and the sells will be done by phone or by some other means.
It is actually planned that on the future there will be an editor application and even e-commerce on the site but that depends on the success of the actual code. So, it is not needed to write those other applications for now but it is good to be prepared to receive them.

No Architecture - "Being Too Agile"

I don't want to criticize the Agile methodology as I think it has many valid points. Unfortunately, many developers justify the completely lack of architecture as being "Agile", and that's why I am using that "Being Too Agile" on this topic.
This "too agile" may happen because there's no discussion at all and developers simply start doing things or because there are discussions focused on the wrong things before starting, maybe more focused on how to name things (like private members and database tables) instead of really focusing on code reuse and the evolution of the application. That is:
  • A database is immediately created using the database the team is most used to (for example, SQL Server). Two tables are created (Category and Product, independently if they have a different name by their standards). At this moment, the team is minimalist, so there's only the Id, IdParentCategory (that can be null) and the Name for the Categories and Id, IdCategory, Name and Description for products. No other tables or columns. Obviously, some fake data is filled;
  • The web application is created with a single form. To "reuse" code, a DBHelper class (which is completely bound to SQL Server and has only two methods: ExecuteNonQuery and GetDataTable) is copied from another project and the actual connection string, which happens to be hard-coded in it, is changed to the new database. In the main form there's a tree view for the categories and list box for the products. The base categories are loaded on the constructor (using the DBHelper class to return a DataTable and then iterating it to populate the tree, in the format "Id - Name") and when clicking in any category, the category Id is extracted from the item's text and used to generate the query of products and sub-categories, by concatenating the id to the base queries (hard-coded on the events).
As you can imagine, all the code is in the form's code behind and the only class added to the project is the DBHelper class.
To me, this is the complete lack of architecture but some people (in particular managers that took only one or two programming lessons but believe they know much more than they actually do) can see only good points, like:
  • Everything that happens on the form can be found by looking at the form's code behind file. There's no need to navigate many "layers of files", there's no complexity to find the implementation of an interface or anything like that;
  • As the DBHelper is built for a specific database, it avoids the "slow" virtual calls;
  • Changing the code to use another database is possible, it is enough to change the DBHelper class source code;
  • The application is up and running pretty fast;
  • Junior developers can maintain the application.
The bad points? Probably none to someone who completely agrees with all the "good points" I just presented. If that's not the case, then there are lots of problems, but I believe they will become obvious when reading the rest of this post.

Code Reuse... Or Not

The customer sees the application. Obviously he requests changes to the UI, layout etc but I will ignore it here. The application is running and there's no request to change its code or architecture.
Yet, the customer thinks that it would be great to have a native application too, so things can run faster and outside the browser. Let's not discuss if browsers are faster now or that the performance difference will not be noticed. Let's simply accept that a native application will be created.
Can we reuse any of the existing code?
And if we don't consider copying the DBHelper to a new project or copying the hard coded queries from the current web application to the new application, then there's nothing to reuse. It's like we can use the current project as the "inspiration" for the new application by looking at its code to take the parts we want but there's no direct reuse, like importing a library or something.
Considering this is a really small application, it would be OK to do a copy, but now is the moment that the future changes are probably taken into account, and it seems better to change only one place than change two places every time. One of the first considered changes is "what if in the future we should not show some categories (like empty ones) or products (that have a deleted flag, an expiration date or similar)?"
As the developers don't see any way to reuse the UI, they want to reuse the database queries and they think about these two solutions:
  • Create a Queries unit that will contain all the queries of the application, and the same file will be shared by both applications (not copied);
  • Create Stored Procedures to do the job in the database.
And I can tell that in most cases the second solution will be used. There are actually many arguments to go in that direction:
  • Changes to the database will not require change to the applications or recompilation of the applications;
  • Stored Procedures are stored in an optimized way inside the database and some even further optimize themselves according to the use, so they are definitely faster than executing different queries from the code;
  • It is said that stored procedures can avoid SQL Injection because Stored Procedures are parameterized, but this is a half-truth (the procedure itself is parameterized, but code that concatenates strings to do the EXEC PROC is still suscetible to SQL Injection);
  • Independently of the half truth, it is true that direct access to the tables can be forbidden, so a DBA can protect the database by forbidding direct access to tables, avoiding deletes and updates to happen if there are no procedures for such actions and also forbidding a query to take too long because a WHERE clause was not used in a SELECT;
  • There's a standard on the code: To call anything on the database there's an EXEC PROC followed by a procedure name and all the parameter values it needs.
So, this is the direction the team takes.
In this case, it happened early in time as the application only has 3 queries. That means that 3 stored procedures are created and the web application is changed to use the stored procedures instead of doing the direct SELECT commands. It would be terrible if this change happened after having 50+ queries.
For now, consider that the database parameters are not used, so the code is suscetible to SQL Injection but there's nothing an SQL Injection can do to corrupt the database at this moment, as only the stored procedures are accessible as read-only and all the data is public.
With this database change, it is now possible to create the native application and "reuse some code". Even the "Id - Name" formatting for the category is done by the stored procedure now and the code of the application doesn't extract the ID anymore (I will not even explore what will happen if the ID can't be displayed). The stored procedures receive the entire Category text and extract the ID. The developers need to recreate the UI, but the "real logic" of the queries is reused. The only code on the application is to add the tree view item's Text at the end of an EXEC PROC call and to read the results and create new tree view items or populate the list box.

Connectivity Problem

To be honest, most DBAs would never allow the database to be externally visible, but let's say the team is not really working with a DBA. They are simply "solving problems fast" and the fastest thing to do was to allow the native application to directly connect to the database.
It worked on some tests but there are two main problems when used in production:
  1. It doesn't work if a proxy server is required to connect to the internet;
  2. If too many native applications are connected, there's an excessive load on the database server, even if most of the connections are inactive.
To the first problem there's no easy solution, as the database connections simply can't pass through a proxy server.
The root cause of the second problem is that the connection string used by the DBHelper class is using a connection pool. It doesn't matter that the pool only has one connection. Every native application is keeping one connection alive. The connection pool is great on the web application, as the same connections are used independently on the client, but they are terrible on the native application that runs on the clients computers.
To solve this issue the connection pool is disabled for the native application. As the DBHelper is a copy, the change doesn't affect the web application. But now the native application becomes slow... in many cases much slower than using the web application as it loses more time connecting and disconnecting from the database than reading some data.

Web Service

It seems that the only valid solution to keep the database server free and to pass through proxies is to use a web service. The web service can actually live in the same server of the web application, sharing the connection pool.
So, the decision is to create a web service to represent all the methods they are using from the DBHelper:
interface IWebService
{
  string GetDataTableAsXml(string commandText);
}
Yeah, they are using only one method. A web-service like this is clearly not how web-services are supposed to work but this is a work-around to allow the code to continue using a DBHelper class and doing those EXEC PROC calls.
So, the actual job is to create a web-service that executes the received command and converts the data-table to string, and change the native application's DBHelper class to use the service and convert the string result back to a DataTable. The application itself will continue to use the DBHelper class and use the same SQLs, so this is the smallest change possible for the application right now.
It works. Many developers would be afraid to deal with that kind of "architecture" (or will try to kill someone) but it works. Bad things will happen if we can't show the ID on the category, but that's not a requirement right now.

Starting Differently

If you think that the previous solution was terrible, well, I completely agree with you. Yet it shows a problem that happens frequently:
  1. An application is created with a bad or inexisting architecture;
  2. All the changes that come later are required to be minimal (usually by time constraints or because the "architects" don't accept their "architecture" is completely flawed), being more work-arounds than a design/architecture change.
But what would happen if they started differently? What would happen if, since the beginning, it was said that a stateless web-service was required, capable of listing categories, sub-categories and products?
I can see that a web-service interface like this would be written:
interface IWebService
{
  // Gets all the categories from the given path.
  // To get the base categories, pass null or an empty string.
  // Returns only the sub-category names. 
  // To create the full path, use the previous path + "/" + categoryName.
  string[] GetCategories(string path);

  ProductInfo[] GetProducts(string path);
}
And the ProductInfo will have all the info for the product that's not already present on the call. That is, it will not have the category path, as it comes on the request, but it will have Name, Description and any new column that may be necessary to present the product.
I am not going too far with the changes, so keep the idea of that DBHelper class and string concatenation when dealing with the database.

The applications

In this case, the Web Application can either use the web-service as an external service or it can invoke the web-service implementation directly, after all they are on the same server.
The client application would use the service from the start and the problems related to direct database access would never exist.
Considering there's only the service accessing the database, it is possible that the stored procedures are never created (remember that the stored procedures were created in the other situation to share some code and in this case the service is shared). So, all the queries can be part of the web-service itself.
Of course there's a big difference from before: There's no IDs being send from the client to the server anymore. The paths are actually created combining the selected category with all its parents (code that is probably going to be copied in both applications, as it is not that big). This also means that the service will probably split the path to find each category ID, executing much more queries when a 10th level category is used.
If such navigation of the categories becomes a problem, the future optimizations will probably include:
  • Caching results on the service. At least while the database is read-only, any kind of cache that avoids new database calls would be great;
  • Creating an extra table with the full paths and the category ID. So, it would be possible to do a single query for products or sub-categories with an inner join to that new table, which can use an equals comparison for the full path;
  • Simply putting the full path as a new field in the product and category tables. This is probably the solution that uses most database space, but it avoids a new table and the joins.
The shared problem with the first case is that by using that DBHelper and string concatenation it is still possible to suffer SQL Injection.
As my purpose is not to discuss about SQL Injection but different architectures, I want you to think about this:
Can you see how the data related stuff was written in a completely different manner simply because we started with a stateless web-service?
I am not saying the applications are much better now. They still have the events written in the code behind, using foreach over the results to create new tree view items, using the data from the UI components to get the category paths and completely ignoring design patterns like MVVM or MVC.
Yet, the decision on how to write the database is different. The queries executed are probably going to be different. And the applications don't have any database query anymore (even if EXEC PROC doesn't show what happens internally, it is a database query). Actually, it is even possible to write a web-service that never access a database, using a XML file or a completely different thing and the applications don't need to change.

The Basic Object-Oriended Approach

Both solutions presented until now are completely different from the most basic object-oriented (OO) approach.
Anyone who knows the basic from object-oriented programming will naturally think about two classes: Category and Product. I am not talking about database tables. I am talking about classes.
The Category would probably have these members:
static SomeCollectionType<Category> BaseCategories { get; }
static Category GetBaseCategoryByName(string name);

Category Parent { get; } // Can be null
string Name { get; } // Can't be null

SomeCollectionType<Category> SubCategories { get; }
SomeCollectionType<Product> Products { get; }
// I will soon discuss the SomeCollectionType

Category GetSubCategoryByName(string name);
Product GetProductByName(string name);
And the Product will probably have these properties:
Category Category { get; }
string Name { get; }
string Description { get; }
// Any other property that seems necessary, like Price, Picture etc.
Thinking about classes to represent Category and Product but also forgetting about a better architecture, developers could simply implement the SubCategories and the Products properties to call the DBHelper.GetDataTable method and make these classes completely bound to the database. There's a big chance they will also expose an Id property.
Also, the use of real properties for the collections or methods like GetSubCategories() and GetProducts() and the result types of array, IEnumerable or something different will greatly depend on how well developers know the Object Oriented principles, how much they will tie the result types with the actual implementation and also how well they follow the .NET guidelines.
  • The use of methods is recommented to let it clear that the action may take time, but this is kind of binding it to an implementation (what if all the items are in memory already?);
  • The use of properties is the most common way to represent child items in most cases, but it lacks the information that a slow database call may happen;
  • I am not covering it in this post, but maybe a Task<SomeCollectionType<T>> should be used as the result type to support asynchronous implementations;
  • Some developers may return lists simply because when reading the database they don't know how many records are there and putting things to a list is their "natural" choice. Maybe they don't know any good practice telling not to return modifiable collections, maybe they don't care as a new list is created each time, so there's no problem if the receiver changes the result;
  • Some developers may return arrays because they are following some pattern like the Reflection methods. People can insert new items to a list, not to an array (but they forget they aren't read-only and items can be replaced);
  • Other developers will use a ReadOnlyCollection because they know the result is not expected to be modified;
  • And others will return IEnumerable either because they learned it is the right thing to do, either because they want to use yield return and avoid pre-loading all the records.

No ORM

Notice that here I am not talking about using an ORM. An ORM would probably generate a similar set of classes, but that would be required by the ORM to map to the database and that's not my focus. I am simply talking about the basic OO idea of having objects that can contain data and behavior.
It would be possible to put these 2 classes into a shared DLL and end-up doing a web application and a native application that are pretty similar to the first case. That is, the native application would still have direct access to the database by using a DLL that has direct access to the database.
The biggest difference is that we will always see Category and Product as objects. Then the objects will do the database job. This would probably change how things are put into the Tree View, so either the objects are put into the Tree View directly and a very base data-template is used to show the Name, or they are somehow stored in a property like Tag to be accessed later without extracting paths or ids from the tree view items.
It would also be possible to have a case similar to the second one, creating a web-service on top of these classes and writing the applications on top of the service. But then the OO approach will very likely only exist inside the service. For the two applications things will stay as a stateless service, not as an Object-Oriented approach.

Object Oriented without Limitations

Wouldn't it be better if things started with the object oriented approach, allowing both applications to deal with Category and Product objects, but without the problems?
That is, the web application could use the objects directly and those objects will access the database. The native application will also be coded as if it was using those objects directly but communicating to a web-service.
So, is this possible?
And the answer is yes. And this is what I mean when I talk about creating an application architecture without considering limitations. I don't need to think on stateless objects. Being stateless is a communication limitation, not part of the application architecture.
So, the only "constraint" I use is that to support all the different scenarios I must start things with interfaces, not classes.
That is, I will have an IProduct and an ICategory. These interfaces could look like this:
public interface ICategory
{
  ICategory Parent { get; }
  string Name { get; }

  IEnumerable SubCategories { get; }
  IEnumerable Products { get; }

  ICategory GetSubCategoryByName(string name);
  IProduct GetProductByName(string name);
}
public interface IProduct
{
  ICategory Category { get; }
  string Name { get; }
  string Description { get; }
  // Any other property appropriate for a product here.
}
As we can't have static methods on interface, we will need an "entry point" to get the base categories. This can be another interface:
public interface IBaseCategories
{
  IEnumerable Categories { get; }

  ICategory GetCategoryByName(string name);
}
Having these interfaces we could write the two applications without any dependency on specific implementations. Those interfaces can be implemented in completely different manners, so we can have implementations that load things from the database, that load things from XML, by using a service and why not a test implementation that simply instantiates two categories with two products each directly? This is what will probably happen when testing the applications for the first time. Instead of dealing with an actual database connection, we simply make things work on top of the interface with a fake implementation.
Of course, at some moment we will need to write an implementation that uses the database (which can have Id properties too) and we could end-up making the bad choice of using that implementation on the native application. But replacing that implementation with one that uses a web-service will only require a one-line change to instantiate a different IBaseCategories implementation (and if done really correctly could avoid recompiling the application, but I am ignoring that part of the architecture for now).
So, up to this moment, the most basic architecture, that I consider to be free of implementation problems is:
  1. Think about all the classes you need, like you will do when writing an UML diagram and, to guarantee that you are not bound to any implementation detail, write everything as interfaces;
  2. Actually, there's no item two. As long as the interfaces represent the right behavior and relationships between the objects, everything is OK at the architecture level. The applications can already be written on top of the interfaces.

Moving problems forward?

Having the interfaces first means that we can have any implementation. But we will need to write the implementations, right? And those implementations can have all sort of problems like they had before. So, aren't we simply moving the problems forward?
The answer to this question is something between yes and no. If we blindly implement the interfaces we can surely have an implementation with all the problems we had in the other scenarios and to fix them we may need to completely rewrite the it. Yet such a "complete rewrite" of the implementation doesn't require a change to the applications.
In fact, it is even possible to request that another team writes the implementation of these interfaces and they aren't required to have the applications at all as long as they have the interfaces, so it is guaranteed that the rework is going to be smaller.

Stateful and Stateless - The False Assumption

"We can replace implementations but we can't use a stateless web-service. These interfaces are stateful. Everybody knows that stateful services don't scale well."
This is probably the killer argument to avoid starting things with the more Object Oriented interfaces. And this is actually false argument.
It is true that if we use the default .NET remoting the products and categories referenced by a client need to be kept alive on the server for as long the the client can use them or else new requests to use those objects will fail. Worse than that, it is the same computer that must answer the requests as all the "object ids" known by the client are server's in-memory object ids.
If we use WCF we simply can't expose these interfaces directly as services because they aren't stateless.
Yet, these are framework-specific limitations. Instead of using in-memory IDs the framework could very well be sending the actual database IDs or even the paths as the information, being capable of reloading the objects if needed and also allowing different servers to answer new requests.
Want a proof of this?
It is possible to create a stateless web-service and implement it to call these "stateful interfaces". It is possible to create an implementation of the object oriented (stateful) interfaces that store the paths and use that stateless service for the calls.
That is, the application can be using the real implementation directly or it can using objects that hold paths and redirect the requests to a stateless webservice, with in turn is implemented to use the stateful objects to do the job.
"OK, it works but it is a lot more work to make it run properly and all the extra work makes it a bad architecture."
Yes, it is a lot of work... if it is done by hand. A better framework will do that transparently or, event better, will simply work differently and avoid the problems altogether.
My purpose here is not to say that we should avoid stateless services or that we must always lose time creating our own frameworks. My purpose is to show the difference between applications written without any architecture in mind, applications written with an architecture in mind, but having the architecture shaped by specific technology/framework limitations and applications that have the architecture made before considering technology limitations.

Risks

An architecture unbound from technology limitations is not always a good thing. Some limitations can't be avoided and others would require so much effort to overcome that it is better to accept them. I can say that at this moment the biggest limitation I see when designing any interface is the sync/async dilemma. An interface shouldn't expose implementation details but being synchronous or asynchronous is an implementation detail that affects the signature of the methods.
To me, that's the kind of technology limitation that we must consider when doing the architecture. If we consider only the most versatible design, without considering performance, it is probably better to make all interface signatures asynchronous as it will support all the cases. It is always possible to give a synchronous result when implementing a method with an asynchronous signature. The opposite is not always possible.
Yet, making absolutely all interfaces asynchronous is a performance killer. So, it becomes a matter of choice and the expected uses/implementations of the interfaces and, in some cases, it is even valid to have both synchronous and asynchronous methods that do the same. I hope this kind of problem disappear in the future.

MVVM, MVC, ORMs and everything else

Up to this moment I didn't solve many problems. In all scenarios there's code directly on the form's code behind and I clearly left all the implementations that access the database susceptible to SQL injection. So, I am doing a terrible architecture, don't you think?
To be honest, I left all those details untouched on purpose. At this moment we have three main areas: UI, the job to be done and the abstraction that let the other two talk. In some sense we can say that this is a kind of MVVM or MVC, but it is not exactly the same. In MVC and MVVM, all the layers are implementation layers. There's no real abstraction.
In the latest solution I consider the UI and the job to be done as the implementation and the abstraction (the interfaces) as the real architecture. That is, it doesn't matter if you use MVVM correctly. If your Model is bound to SQL Server and nothing else, it would be a problem to make things work through a web-service or similar. If you have the right abstraction first, then it is pretty easy to do that change.
Yet, it doesn't mean that you should avoid MVVM or using an ORM. In the end we always need an working application (or two, as the web application is not the native application), and having a good architecture is only the start. When going to implement things, if an ORM will help the team write easier to read queries and avoid SQL Injection, they should go for it. If the code behind is a problem because the designers don't know what to do with it, then go for MVVM. Only remember that those are part of the implementation. Maybe you can consider them sub-architectures as they will greatly influence the code that's going to be written, but the main architecture is built on the purpose of the application. MVVM, MVC and ORMs exist independently of the applications and should not be considered the architecture on their own.


Ref From 
http://www.codeproject.com/Articles/889468/Software-Architecture-Examples

at No comments:
Labels:
Subscribe to: Posts (Atom)

What should you required to learn machine learning

  To learn machine learning, you will need to acquire a combination of technical skills and domain knowledge. Here are some of the things yo...