java - Strategies for designing a database (being accessed by hibernate) which will have a lot of archivial data -
I am developing an application that will be integrated with thousands of sensors that send information every 15 minutes intervals. Assume that the data for all sensors is the same as the format. What is the best strategy to store this data, so that everything can be stored (accessible) and due to the large size of mounting data it has no negative effect.
This question is related to general database design. I suppose, but I would like to say that I am using hibernate (with spring rupees), so maybe something is already addressing it from there is.
EDIT: The sensors are dumb and are not far from the shelf, it is not possible to expand them. All information is lost in case of network outage. Since the sensors work on GPRS, this scenario will not be possible (as the GPRS provider is not good at here in Sweden, but yes it can go down and no one can do anything about it).
A amount of penalty was the most important, and provides easy spring to work with prototype codes based on STVMM. Let's assume that you have information about sending 10,000 sensors every 15 minutes.
In order to perform better on the database side, you may have to divide your database by date / time, sensor type or category or some other factor, depending on how you will ask for your data.
The other bottle neck will be your Java / Java EE application. It depends on your business, such as all 150,000 sensors send information at the same time? And to follow the architecture of your Java application, you need to read the article on high scalability and performance.
Here is my recommendation for Java / Java EE solution.
Instead of a single, cluster of applications that receive a data.
is a controller application that controls the link between which the sensor sends data, in which the application has a frequency of clusters. The application instance can pull data from the sensor or push the sensor data into an application example, but the controller is the one which is related to the application example, which is connected to the sensor set. This controller should be dynamic in such a way that the sensors can be connected or removed or updated, as well as any time the cluster can join or apply. Make sure your administrator has some failures.
So if you have 10,000 censors in the cluster and 10 instances of the application, then you have 1000 sensors linked to an application at any time. If you still want a better performance, then you can call 20 applications of application in the cluster and you will have a 500 to 500 sensor application frequency associated with it.
Examples of the application can be hosted on the same or multiple machines so that the horizontal horizontal scalability is achieved vertically. Each application example will be multi-threaded and has a local firmness. This bottle reduces your transaction response time from the neck to the main database server and not on the server. This local firmness can also be a SAN file (s) or local RDBMS (such as Java DB) or even MQ. If you live in the database locally, you can use Hibernate for that.
Asynchronous data is taken locally firmly to the main database. It depends on how you have continued the data locally. If you use file-based firmness, then you are separate Thread is required which reads data from file and inserts in main database store. If you use a local database, this thread can use Hibernate to read data locally and insert it into the main database repository. If you use MQ, you can apply threads or different applications to move data from QE to main database repository.
There is a reflection in this solution that there will be some interval between the sensor, this data is displayed in the main database.
The advantage in this solution is that it will give you high performance, scalability, and failure-overs.
Comments
Post a Comment