Intermittent Back-end Amazon S3 Errors

February 15, 2008 – 9:49 pm

Earlier today we experienced intermittent failures reading from, and writing to, the Amazon Simple Storage Service. This would have affected some account creations and the reading of writing of documents to KnowledgeTreeLive. KnowledgeTreeLive does attempt to retry writes to the S3 repository and, if it fails, will notify the user of the failed upload of a document.

It is understood that Amazon has resolved the issue and service has stabilized.

System Status 21:12 UTC 15 January 2008

January 16, 2008 – 11:14 am

We appear to have sorted out our JIRA issue tracker for now. We are encountering issues with it periodically and hope to have a more permanent fix in place soon.

System Status: 09:11 UTC 15 January 2008

January 15, 2008 – 11:12 am

We’re having intermittent issues communicating with our ticketing system from within KnowledgeTreeLive. This will only affect access to the ticketing system for bug reporting.

We’re working on the problem and hope to resolve it shortly.

Refresh Completed - 07:00 UTC

December 20, 2007 – 9:04 am

We have successfully completed a refresh today! This refresh sees the base system migrated from KnowledgeTree 3.4.3 to KnowledgeTree 3.4.5. Various stability fixes, improvements and bug fixes that are specific to the Live offering were also included. Enjoy!

KnowledgeTreeLive is out of Maintenance Mode (19:00 UTC)

December 17, 2007 – 9:26 pm

What was initially assumed to be a DDOS attack turned out to be a corrupted NFS mount. All services have been restored to normal operation.

KnowledgeTreeLive is Currently in Maintenance Mode (16:30 UTC)

December 17, 2007 – 6:33 pm

We’ve taken KnowledgeTreeLive into maintenance mode and are investigating a possible distributed denial of service attack (DDOS) against our cluster. Our systems engineers are currently performing virtualization ju-jitsu within the cloud and we expect to restore service shortly.

System Status - 09:05 UTC

December 13, 2007 – 11:05 am

Maintenance has been completed successfully.

System Status - 08:05 UTC

December 13, 2007 – 10:03 am

KnowledgeTreeLive will be put into routine maintenance mode for five minutes between 08:30 UTC and 09:00 UTC.

System Status - 16:10 UTC

November 20, 2007 – 6:28 pm

We experienced a temporary database connectivity issue. No loss of data occurred. Only one of our database servers was affected by this, and we were able to bring in another server that was on hot-standby.

We are putting additional safeguards in place to ensure that this does not happen again.

System Status - 12:15 UTC

November 20, 2007 – 2:38 pm

Maintenance was completed withing the maintenance window period, and all services are back to normal operation.