The Critique Circle Blog

The CC Blog is written by members of our community.
Do you want to write a blog post? Send Us a blog request

Menu
  • View all blogs
  • Go to thread
Mar
16
2020

Rewriting CC [part 2] - The Plan -- by Jon Bjarnason

As we discussed in the previous blog we need to move away from classic ASP and into a modern web framework so that I can continue to evolve the site. Choosing a web framework often isn't easy since there are so many to choose from and I wouldn't want to get stuck on the wrong horse again.

I have been writing RESTful webservices in Python on the Flask microframework for a few years now and I think this would be a good fit for CC.

I have a lot of experience in working with Amazon Web Services and that is the platform that I will use for the the new CC website. This platform gives us a lot of freedom on how services are split up and allows for elastic scaling based on usage, giving a good balance between performance and cost.

The stack will sit on a Linux system with a Postgres database. The advantage of moving from windows/ms sql to linux/postgres is that there are no licensing fees and the operational costs will be slashed by half.

The problem is, however that there is very little common ground and I will be required to literally rewrite every single line of code. As we saw previously, this is well over 100 thousand lines (we're talking Harry Potter in VBScript here).

After mulling it over for a while and doing some tests I've concluded that the best course of action is to rewrite individual systems and run a new website alongside the old one for a while as I progress through the rewrite.

The first thing that needs to happen is to reduce the current website down to its fundimental parts and get rid of any features that are not used much. I want to do this first for a couple of reasons. Firstly, because these extra pieces of functionality are in my way when I'm breaking down the work that needs to be done and figuring out which part goes where, and also because I want to see if any of the features will be particularily missed by the members so that I'm not in the unfortunate position of having taken some toys away at the time I open the doors to the new site, which will cause bitterness when we should be celebrating.

Among the features that I want to remove are the following:

Spell checking

I hope no one is using this feature anymore since the spell check in current browsers is so much better.

Quick text

I personally use this for quick replies but I could just as well use a browser extension.

In-site bookmarks

This is something from the early days of CC and I don't see any reason to have this anymore. People can just use browser bookmarks.

Hangman and word morph games

They are not used much but maybe I will reimplement them or add some other games on the new site.

Private Forums

The way people communicate has changed. Currently these are hardly used at all except to discuss stories in private queues. Forum discussion of stories also isn't done much anymore but if needed I will implement a single story comment system instead.

World map

This is a fun little feature that I might re-implement later but not worth the effort now.

Reminders

People don't need reminders on CC, we have google calendar for that.

Statistics

The stats pages use a graphic library that I will likely not be able to migrate to the new server on aws. Probably at some point I will be able to make even better pages on the new site using some interactive library.

Non-inline crits

In the new site there will only be one critique type, which will be inline, possibly including in-paragraph commenting. To start off with I will remove the classic and template crit type options (the latter has already been removed) and new crits will always be inline. You will be able to submit a 'classic' crit by just leaving a comment in the 'comments before' box.

Other tools

I will probably move over the tools one by one towards the end of the development. I think the outlining workshop will probably be at the top of the list to get back in. The manuscript progress and word meter will also be high, but I'm not sure about the rest since most of the tools will require a lot of work and most of them don't get much use.

 

There are a few other little features that I will turn off soon and I know it will annoy some people, but believe me whan I say that it's for the greater good. Please let me know in the comments section though if you really miss some features and maybe they will make a comeback in the new CC somewhere down the line.

 

Forums

I am hoping to be able to use some off-the-shelf forum solution instead of rolling my own. This would save me roughly a quarter of the work overall, but it might cause forums not to be as tightly integrated into the rest of the site as they are today.

On the other hand, integrating modern forum software will give us much richer and nicer forum experience than we have today so I believe we will come out on top.

There are several instances where we have deep forum integration with other features, such as story comments and blog comments. I will need to cut these ties and make the forums a distinct application, separate of the core CC, but sharing authentication. The forums might even move to community.critiquecircle.com.

Because of this story comments and blog comments will likely not be able to utilize the forum system.

The Plan

We need to move from a monolithic classic ASP website written in VBScript to a modular web application written in Python utilizing cloud architecture. In short, how do we go from this:

to this:

The exact moving parts here are not very interesting to you guys but the important thing is that this is an entirely different architecture with a lot of things that need to be migrated or rewritten.

Although I would like to do it all at once I'm simply not able to. This will me a multi-month operation for sure and I will need to be in a position to allow both the new and old code to work at the same time.

To achieve this I will need to ensure that both sites are able to use the same database. AWS allows me to do this by using their hosted RDS service, taking the database out of the application box and making it available to several services. This is fundimental to the strategy.

Keep in mind through all of this that my immediate intention for this rewrite is not to improve the functionality of CC but just to get it done. We will look into improvements later, once we've transitioned 100% into a modern web framework. Therefore, after the bulk of the work is done, if we're lucky, CC will function much the same as it does today.

Phase 1 - Simplify the application

There is a lot of code in CC that has been built up over the years that isn't doing much and might be removed. There are also quite a few features that are not used much and will be time consuming to rewrite. I have already started removing unused tables, code and features and slimming down the database. This will make the move to AWS smoother. Sadly this means that some users will lose certain features that they like, at least temporarily.

Phase 2 - Move CC to AWS

Once CC is slim and thin I intend to move it as-is from the current bare-metal hosting provider to an AWS EC2 instance. I will be splitting up the DB and the application, but other than that there should be no change.

Once this is done we should have something like this:

I've already done some tests that look quite good. We should be able to get this part done in a reasonably short time, which will take care of the immediate problem of losing the current hardware.

Since the database will be on a dedicated machine managed by AWS I have hopes that performance will be more consistent than what we experience currently,

Phase 3 - Migrate to Postgres

Now we can start to migrate the database from Microsoft SQL Server to Postgres which is a much easier database engine to work with from the Python/Linux side (plus it's free). This work will involve moving the stored proceedures into code or postgres functions. There is luckily not a lot of high level functionality in the database but there is some. During this I will further simplify the database architecture and throw away unneeded functionality.

Since we will still be running the old vbscript website I will need to move the db functionality into VBScript even though it will be discarded soon. We also need to change some table structure to conform to postgres. This will be a reasonable amount of work but some perliminary tests indicate that this can be done and that classic vbscript can use postgres as a backend.

Again, once this is done CC should be running as it did before, but with a whole other engine under the hood. Now I will be all set to begin the actual transition into a new architecture.

Phase 4 - Bootstrap the new website

Now I can start to build the first 'hello world' application of the new website. This will involve setting up development and deployment pipelines, setting up docker, lambdas and redis stacks, choosing modules and web stack pieces, getting OpenID Connect up and running and getting everything to work smoothly on AWS. This will be completely behind the scenes and with minimal changes to CC itself.

Once this is done I will be able to start writing features against the common datastore. We will likely have two websites at this point, the old one in classic ASP and a new one in python. Some of the links on the website will lead to the old website and some to the new. The transition between the two will hopefully be rather seamless.

Phase 5 - Reimplement core functionality

Now I will start to build up a brand new website from scratch, maybe hosting it on develop.critiquecircle.com. This website will use the same backend database as the old site and should be usable by members. Once features start to stabilize I will probably encourage users to transition to the new site so that we can see how it behaves. However, when you enter many of the sections of the site such as stories, forums or tools you will jump to the old site since the features don't exist on the new one yet.

I have not decided in which order the features will be implemented, but login, settings, member info, blogs, admin functionality are probably high on the list.

At this point the architecture looks something like this (removing a bunch of things for simplicity)

Basically we have two independent websites using the same database with some features being delivered by the old one and others by the new one.

Phase 6 - Reimplement stories

This is the big one. In this phase I will rewrite stories, queues, novels and crits. There are so many features, nooks and crannies in these systems that will need to be worked on that I fear this will take a long time.

It's very possible that some features will not make it in on the first pass and from the user's perspective we will be regressing in terms of functionality. Hopefully by the time we're completely done the stories section will function much like it did before.

Phase 7 - Move forums into an off-the-shelf module

Now I will deprecate the forum functionality in favor of a forum package that I will hopefully not need to implement myself. The forums will probably look very different from that they are today and some features will certainly be lost. However, we should get much snappier and more modern forums as a consequence.

At the end of this phase only the tools section will remain on the old website and those will be moved into legacy.critiquecircle.com. I'm not certain how many of the tools will be moved into the new site at all, it will have to be seen.

Phase 8 - Address remaining functionality

Any remaining functionality will be moved into the new site or completely removed. Some features might then be reimplemented on the new site later on and others will disappear forever.

We can now shut off the old website and will the transition will be complete. However, there will still be a lot of work done.

Phase 9 - Refactor functionality

Once all the previous phases have been completed critiquecircle v2.0 is running and we have removed the legacy code. However, the functionality is much the same as on the old site, the tables and columns are named the same and there are a bunch of hacks in place to make things work the way they used to.

Now that the legacy website has been removed I will need to work on transitioning to a modern architecture. This work will probably last a long time, but we will be in a good place to do it having all the tooling and framework in place.

At this point in time we will finally see the light at the end of the tunnel and start to get in some much needed improvements.

Posted by Jon Bjarnason 16 Mar at 00:17
Do you want to write for the Critique Circle Blog? Send us a message!

Responses to this blog

Tonin 16 Mar at 07:58  
This all sounds good to me, and I appreciate your continuing dedication to keeping the site alive.

Im curious about the login system, since you mentioned transitioning to third-party authentication. Will there still be a stand-alone alternative? I imagine were a minority, but there are some folks who dont have (or want) Google or Facebook accounts.
Nonnib (administrator) 16 Mar at 09:08  
Im curious about the login system, since you mentioned transitioning to third-party authentication. Will there still be a stand-alone alternative? I imagine were a minority, but there are some folks who dont have (or want) Google or Facebook accounts.
Yes, we would retain the ability to have local cc accounts.
Lfrankt 16 Mar at 12:21  
I heartedly endorse the move to a new architecture to make the CC site more efficient and expandable. One thing I'd love to see is the ability to test the development version at certain release stages to provide you feedback on the site's UX design and performance. As a former member of a software team that provided developer tools for writing cell phone apps I've plenty of experience in UX design, rapid development, and meeting project goals. Keep the code manageable, do it a functional section at a time, and be thorough in testing. It takes a lot of work but everyone will appreciate the result in the end.

Can't wait to see the new modern CC site.

Lmdewit 18 Mar at 01:44  
I am in favor of your plan. It makes sense to me to have the basic functionalities then upgrade as you see fit.
From all the things you want to remove, I'm okay with all but private forums... though that's something perhaps only I and a couple of friends used a lot (not much anymore, alas, so I'm guessing we'll have to find other ways to discuss our own stories).

One question though... are you planning to keep ALL the stories and crits on the site? It seems a mammoth effort as you mentioned and wouldn't it make sense to just upgrade perhaps the last couple of years and let the users know? Would that cut part of your effort?
Brinker 18 Mar at 03:17  
This is all great, and I look forward to the new site.

Could you possibly put a link to this up in the "news" section? I generally completely ignore the blog and related forums, so nearly missed this until I saw someone else comment on it. I imagine there are a number of other people who haven't noticed it, either, which is unfortunate since it's pretty noteworthy.
Ellies 18 Mar at 04:43  
I have a request:

When a person goes to revise a story, there is an option below the writing area that asks if you want to "paste text from word document"

If I accidentally click on it while working on my revision, the entire submission disappears.

Would it be possible to ask if I really want to paste in text? Kind of like when you confirm a selection like deleting a story, or some other permanent action? Does that make sense? It causes me to freak out, even though I can click "back" and hope it reappears without wiping out the story.....

Thanks !
Vkkerji 18 Mar at 05:49  
Jon,
When I read your first part, I thought you will be creating an altogether new project and once it's complete, you would be hosting it as a new CC website. In this post, your method to implement the CC one portion at a time is plausible. I think this will enable you to learn from the users about what went right and what needs to be modified.
I really appreciate your detailed technical explanation and I am sure you have thought it well before you start implementing it. I wish you the best and look forward to seeing the new pages functioning!

Regards,
Vijay
Tyswan 18 Mar at 10:46  
Hi Jon,

It looks like a gargantuan undertaking . I don't envy you.

Non-inline crits
In the new site there will only be one critique type, which will be inline, possibly including in-paragraph commenting.
I've used a few online crit sites, and I prefer CC to them all.

I comment on particular sentences, but the bulk of my crits are reader reaction and big picture comments. These are best applied at a paragraph level. I find it frustrating on other sites to find the right place to put these kinds of comments, which is why I prefer CC. Does the note about possible in-paragraph commenting address my needs?

Oh, and I'm one of the people who uses both private forums, and quicktext . But I understand that other people don't use them, so you have to cater to the 99% .

I do like the idea of a user status/feed that you mentioned in your last post.

Lastly, I really like the CC emoticons . I hope you won't change to another emoji set . I've gotten so used to them , and expressing myself through them that I sometimes wish they were the same on other sites.

Anyway, thanks for all of your hard work, and good luck with the huge task ahead. You might want to open up a forum for brainstorming some of the technical and design aspects with those of us with Software Engineering experience, because it can be daunting trying to make some of these decisions on your own. After all, many of us here are expert business users and understand the functionality of CC very well. And it does seem that there are a lot of IT professionals on this site. Obviously you don't want to design by committee, but sometimes thrashing ideas out with people who know exactly what you're talking about (from a user requirements perspective), can help to clarify your own ideas.

Cheers,
Tys.
__________________

Tyswan Slater
writer ~ illustrator
www.tyswan.com

Andymather 19 Mar at 20:09  
Please let us know if there's anything we can do to help, other than complain that our favorite toys are gone :{D Well woth it in the end! Many thanks.
Andymather 19 Mar at 20:09  
Please let us know if there's anything we can do to help, other than complain that our favorite toys are gone :{D Well woth it in the end! Many thanks.
Rhitakorrr 23 Mar at 20:02  
Very cool. I'm looking forward to this. As far as forum software goes, Discourse is open source and pretty popular nowadays. I think it's rather nice.
Dougp 24 Mar at 02:18  
Thanks for sharing the architecture, the challenges, and the phase plan. Of course, most authors don't have any idea what you're talking about, but that's okay. You're being transparent so that when something breaks (it will), we'll all recognize the huge task you're undertaking and not complain (well, some of us will still complain, but just appease us with a free credit every now and then and we'll be happy). Best of luck - it sure looks like you've done your homework.
Rellrod 24 Mar at 02:38  
Oh, that's where my bookmarks went. No problem, I can live without 'em.

I love listening to your detailed description of the changes, though I only understand about 25% of what you're saying.

Again, thanks for your immense time and effort on this project. It's creating something wonderful for all of us!

Rick
Onalimb 24 Mar at 19:35  
The changes sound great. I'm looking forward to seeing them. Thanks for taking it on.

Could you perhaps create a new thread in the "About Critique Circle" forum to keep us apprised of scheduled changes? For example, before you remove a function/data, it would be great if you could give us a heads-up, so anyone depending on it has time to act.




Nonnib (administrator) 25 Mar at 18:22  
Thank you all for your comments, it's much appreciated.

I'll set up a thread like you suggested Onalimb.
Aventurist 28 Mar at 17:37  
I hope this doesn't sound too dim. I recently built two sites, one for myself and one for my company. It had been many years since I had written a site and I was shocked to discover I didn't need to write it in HTML any more. Instead, I built it using Concrete 5, a CMS, an alternative to Wordpress. So, I am wondering if most of the rebuild could be done using a CMS instead?
Nonnib (administrator) 29 Mar at 10:11  
Thank you for that feedback. We have a lot of dynamic functionality here with regards to stories and critting that a cms won't be able to handle but for other parts like faq and blog and such it can make sense and it's something that I am evaluating.
Ceronj 30 Mar at 10:25  
I've read your plan. As a software developer myself I can say it's ambitious, leading edge, and well thought out. I'm in a Java/Kafka shop so I haven't done any work with Python. But it's becoming pretty popular. I wish you luck.

BTW, do you run automated testing on Jenkins?

-J
Jeff65 31 Mar at 18:42  
I really appreciate your dedication to keeping the site alive and your willingness to undertake all that wor.
Nonnib (administrator) 31 Mar at 21:00  
Much appreciated. As for Jenkins, probably not. I'll most likely go for travis or azure pipelines for CI.
Doree 1 Apr at 17:55  
I read. And then I reread and it's a good thing that I can follow directions because I'm good over here and seeing the finished work. Thank you for this and all of that, too.

Wallaby 2 Apr at 14:29  
Will we be able to use gifs on the new website?

giphy.com/gifs/14cNSEhRyTX8Ag/html5


Boborama 2 Apr at 19:19  
My two cents on this an Android Developer that started out as a web developer almost 25 years ago. It seems from your plan, you will migrate the old code to the new stack, make backend changes (database) and then migrate the data and then change the code to the new schema and then redeploy while you build the new code which once that's ready, you'll just throw away the old code with the changes.

This is a lot of work that will happen that will just be thrown away. Work is time is money.

Is there a way you can not do that migration, just run the site as-is build it new, migrate the data as needed to the new system? In other words, just not try to get the existing code to run on the new AWS?

It looks like you plan to run both simultaneously (Phase 4) and sunset features as the new ones become ready? I think that's going to be a lot of extra work and it's going to create bugs as you do that integration which takes time from the new development.

Just my two cents. I've worked on a lot of migration projects and there isn't a new solution. But if you're not planning on reusing code (microsoft to python) rather than piecemeal, I'ld build it straight up and turn it on all at once.

Best of luck!

Nonnib (administrator) 3 Apr at 11:11  
Boborama
...
Is there a way you can not do that migration, just run the site as-is build it new, migrate the data as needed to the new system? In other words, just not try to get the existing code to run on the new AWS?
...
Yes, that is how someone might normally do something like this and you're right that it is a bunch of extra work to do it piecemeal but the fact of the matter is that I've made half-hearted attempts at starting to write a new site from scratch more than once over the years but it has always been such a huge undertaking that I have been unable to really get started.
I will be spending many months on this, maybe even a year to get everything done practically as volunteer work and if I was in a silo I'm pretty sure I would give up before I got to the end.
This way, I will be able to release smallish features as soon as they are completed without having to worry about all the other things as well. We're then able to iron out the kinks and get things up and running properly before I move on to the next thing.

So, if I was running a software house with some staff I'd most likely just write a new site with all the bells and whistles and then migrate over, but I am sneaking in an hour here and there while I have work and family stuff so even though it's more work overall it should be easier to spread out as needed.

Doree 8 Apr at 17:33  
I haven't run into any problems and I've been on this site everyday since you started the migrate. To me, I say handle it the way you are doing it, it's working. Thank you for your work.
Vandrelyst 30 Apr at 04:12  
I am sad the private forums are going away. I don't use mine a ton, but it's nice to have. I've used it in the past to share updates about my progress, art that I've drawn to go with my novel, etc. I was actually about to start a thread full of music I listen to when writing my novel, in case any of the readers wanted to check out the playlist. But I saw the notice that it'll be gone soon, so I won't bother, I guess.
Troglodyte 7 May at 20:48  
Hope you have room for the Name Generator, it's the one tool I use regularly. But I'm a big girl, I'll adjust, and having been an administrator on a Learning Management System, I have a tiny idea of what you're up against. Also, I love shiny new things. Good luck!
Vandrelyst 22 May at 17:11  
Is the "most useful crit" and "free pass" system removed? I realized I haven't been prompted to select a most useful crit in a while, and can't even find the option.
Maiacer 27 May at 20:26  
I agree with Tonin. I hate Google and FaceBook and I have had all sorts of problems losing accounts and passwords when I have been asked for third-party verification. I love CC and do not want to leave. Before I donate, I want to be sure that Google or FaceBook check-in is NOT something I will have to deal with in the future.
Attaree 28 May at 06:54  
I'd like to be able to hide my stories and keep my crits visible.
Icedtea3 20 Jun at 20:58  
Would the new website be more aesthetically pleasing?
And also, which website would you log into to say, just crit and submit your own stories?
Attaree 21 Jun at 02:15  
Request: On the crit grade form, I'd like all the good grade buttons on the right side and all the bad grade buttons on the left side.
__________________
I aspire to write redneck, blue collar stories of literary quality.

Respond to this blog

Please log in or create a free Critique Circle account to respond to this blog


Member submitted content is © individual members.
Other material is ©2003-2020 critiquecircle.com
Back to top