Have you ever wondered on how to create persistent datastructures.......?

  Hey recently I came across a problem on how to implement external memory datastructures???........external memory datastructures are just like normal datastructures but operations you make on these datastructures are stored in secondary memory.......   PROBLEM: My server application (assume a dashboard application running on a server)has a binary tree to keep track of the posts posted by the user....as long as my server is up and running...and when the user creates new posts, Iam adding the posts to this tree...every thing was fine till recently before my dashboard became a hotspot on the internet...suddenly I found as the posts increased the tree is growing bigger and bigger and eating up all the RAM space on my sever Ok I still have a 4GB RAM..but its hitting the performance.....what should I do as the person who wrote the program to handle this stuff.....I cannot keep all the 1 million posts in the RAM....ok the OS's virtual memory is taking care of doing this (Virtual memory is doing the swapping)...but its leading to thrashing....ask the virtual memories code is not application specific.....So whats the solution???? SOLUTION: All applications which need to handle huge amounts of data in the application code should have a inbuild PERSISTENT-RUNTIME which does the application specific paging.....I guess the code below will make things clear than words.....   I had this RUNTIME code in all the persistent code which I had written...I guess its very easy to tweak for your application.............Planning to make a opensource project which does better caching etc...... Get in touch with me I'll sell it if you want to buy............the following is the interface....    

#ifndef __Persistence_H__
#define __Persistence_H__
#include
#include

typedef int PAGENUMBER;
typedef void* PAGE;
typedef enum PersistentError{PSUCCESS=1,
        PREADFAILED=-2,PWRITEFAILED=-3,PPAGESIZEEXCEEDED=4} PERROR;
/**
 * The Cacheing structure
 ** This helps in doing the inmemory Lookup
 **/
typedef struct PersistCache{
//TODO: Some one can extend this interface

        bool (*pageExists)(PAGENUMBER)=NULL;
}PCache;
/**** The core Persistence Engine **/
typedef struct Persistence{
        int PageSize;
        FILE *fptr;
        PCache *PageCache;
        //The core routines//
        PERROR (*startPersistence)(const char*);
        PAGENUMBER (*getNewPageNumber)(void)=NULL;
        PAGE (*getPage)(PAGENUMBER)=NULL;
        PERROR (*writePage)(PAGENUMBER,PAGE)=NULL;
        PERROR (*defragmentPersistence)(void)=NULL;

}PersistEngine;
#endif