Kaltura Performance Issues
Incident Report for GSU Technology Systems
Resolved
Kaltura has worked to improve system performance throughout the Fall 2020 semester. The issues we had been observing have been resolved, but please contact help@gsu.edu if you continue to experience Kaltura problems.
Posted Nov 04, 2020 - 10:43 EST
Update
Earlier today, Kaltura experienced a brief service interruption. They have implemented an emergency fix to restore service. Please note the following component updates:

Upload – In Progress – Uploads are overall operational, however, some customers may see sporadic errors during upload.

Transcoding – Update – Kaltura resolved the issue affecting entries that may appear in “converting” status for longer than expected.

WebEx – Resolved – Kaltura identified and resolved an issue related to an issue with converting Webex recordings.

Captioning Sync – Monitoring – The Captions queue is processing normally. Kaltura is continuing to address previous Caption audio/video sync issues.

Please contact help@gsu.edu if you continue to see errors related to these or any other issues.
Posted Sep 30, 2020 - 10:06 EDT
Monitoring
Kaltura has provided the following updates for their on-going issues:

Upload: Kaltura is continuing to implement more fixes to improve upload performance.

Playback: You may experience buffering or slow playback on some entries, most notably on entries that were uploaded close to the time of the attempted playback.

Captioning Sync: Kaltura is working with caption providers to investigate this issue. Captions may be out of sync with video at this time.

We will continue to provide updates as they become available.
Posted Sep 21, 2020 - 10:03 EDT
Update
Kaltura is currently experiencing issues that may cause interruptions for video upload and playback. Kaltura has identified the problem and is taking steps to resolve the issue.
Posted Sep 17, 2020 - 17:21 EDT
Identified
The newly identified playback issue has been resolved, and Kaltura is monitoring the situation.

Regarding on-going issues:

Update - Upload: Some customers may still experience delays with uploads as well as issues with upload functionality, such as progress bar stuck, error messages, and incomplete uploads. To get assistance with a specific entry upload, please reach out to help@gsu.edu.

Update – Transcoding: Users may experience longer than expected processing time to encode entries that have been successfully uploaded.
Posted Sep 11, 2020 - 15:40 EDT
Investigating
We are currently investigating an issue that is affecting playback of Kaltura entries. We will provide updates from the vendor as they become available.
Posted Sep 11, 2020 - 15:32 EDT
Identified
Kaltura has provided the following updates regarding the on-going issues with videos in Kaltura:

Update – Upload: Kaltura is working to identify the cause of problems with upload functionality such as progress bar getting stuck, error messages, and incomplete uploads. For assistance with a specific video upload, please contact help@gsu.edu

Update – Transcoding: You may continue to experience longer than expected processing time for videos that have been successfully uploaded. The root cause has been identified, and Kaltura is working on the solution as a top priority.
Posted Sep 10, 2020 - 08:26 EDT
Monitoring
The file upload issues from earlier today have been identified and addressed. Kaltura is continuing to monitor the issue.

For entries that were uploaded prior to September 7th with status still showing “pending” or “converting”, please contact help@gsu.edu.
Posted Sep 08, 2020 - 17:15 EDT
Identified
Kaltura has identified an error with uploading new media. They are working to resolve the issue now. We will update as soon as we have additional information.
Posted Sep 08, 2020 - 12:38 EDT
Update
Kaltura has provided the latest updates for the on-going issues:

Update - Playback: Kaltura has identified an issue affecting Playback on entries that were uploaded/transcoded on September 3rd. Symptoms include buffering, spinning wheel or dark screen. The root cause has been identified, and addressed.

Update - Transcoding and Upload Delays: The transcoding backlog has been eliminated and you should expect normal processing time. Kaltura is closely monitoring the queue.

Upload Errors & Playback Errors: Kaltura is closely monitoring the system to ensure normal performance.

Update - Captioning performance: Resolved, and the queue is cleared. Kaltura will continue to monitor the queue closely in the next coming hours.

Update - Doc Conversion: Work is in progress.
Posted Sep 07, 2020 - 10:11 EDT
Update
Kaltura has provided the latest updates for the on-going issues:

New – Playback: Kaltura has identified an issue affecting Playback on entries that were uploaded/transcoded on September 3rd. Symptoms include buffering, spinning wheel or dark screen. The root cause has been identified, and they are working to resolve this issue as the top priority.

Update - Transcoding and Upload Delays: Kaltura expects the queue to clear over the weekend and are closely monitoring this situation.

Upload Errors & Playback Errors: Kaltura is closely monitoring the system to ensure normal performance.

Update - Captioning performance: We are still working with Kaltura to find the root cause for the delays.

Update - Doc Conversion: Work is in progress.
Posted Sep 04, 2020 - 15:07 EDT
Monitoring
The errors from this morning have been resolved, and Kaltura is now be available.
Current Status of Open Issues:

Update - Transcoding and Upload Delays: Since the last update, Kaltura has doubled the processing capacity to address the delays. They expect the queue to clear over the weekend and are closely monitoring this situation.

Upload Errors & Playback Errors: Kaltura is closely monitoring the system to ensure normal performance.

Update - Captioning performance: We are still working with Kaltura to find the root cause for the delays.

Update - Doc Conversion: Work is in progress.
Posted Sep 03, 2020 - 09:18 EDT
Investigating
We are seeing API errors when accessing Kaltura. We have escalated this issue to Kaltura and will update as soon as we have additional information.
Posted Sep 03, 2020 - 08:47 EDT
Update
Kaltura has provided the following update regarding the on-going issues:
 
• Outages last week in their data centers were a result of 300% more load than anticipated. This pushed them to accelerate the migration to Amazon Web Services (AWS) that occurred this past weekend.
• AWS migration timing was not ideal with the start of classes falling during this period. Kaltura felt the move would be critical to handle the anticipated increase in usage & volume this fall, all leading to the decision to proceed. 
• The world wide web outage Sunday (and continuing intermittent outages) has caused additional backup in the upload and transcoding queues. 
• The migration is complete to AWS, but they are continuing to work through tuning upload, API stability & further transcoding delays. Kaltura reports that they are at 90% health right now. 
• The migration team identified an issue yesterday that caused API problems. The design issue has been remedied and now they're continuing to process the transcoding backup.
  
Ongoing issues:
• API errors continuing
• Backlog of transcoding and upload delays -- rate of processing is improved, but there’s a large queue of videos to process. Kaltura is closely monitoring and expect the queue to clear in the next 24 hours. 
 
All of this has resulted in a support queue that is immense and backed up as well. Please know that this is the highest priority at Kaltura to get resolved immediately and have a successful experience for the USG institutions and all Kaltura customers.
Posted Sep 02, 2020 - 12:27 EDT
Update
The vendor, Kaltura, has acknowledged performance issues, has taken steps to rectify the issues which includes moving to a new hosting environment. This move will allow them to scale up resources to more quickly respond to usage increases. The vendor has indicated that there is a backlog of videos that must be processed.

GSU and other USG institutions have carried your complaints and concerns about the reliability of their service forward and will continue to advocate for improved services and better responses to issues.
Posted Sep 02, 2020 - 11:25 EDT
Update
Kaltura has seen improvements, but you may continue to experience some issues.

Current Status:

- Transcoding and Upload delays -- rate of processing has improved, and Kaltura is working to finish processing videos in the queue. The vendor expects the queue to clear in the next 36 hours.
- Webcast File Upload and Doc convert -- a fix is expected to be deployed within 24 hours.

Resolved:
- REACH captions -- captions are processing normally. If your videos have not auto-captioned correctly, you may need to request new captions by accessing the video, selecting Actions -> Caption & Enrich, and choosing to order machine captions.

We will continue to monitor these issues with the vendor and provide updates as they are available.
Posted Sep 01, 2020 - 17:22 EDT
Update
Kaltura has seen improvements, but you may continue to experience some issues.

Current Status:
- Transcoding and Upload delays -- rate of processing is improved, but there are still many videos queued for processing. Kaltura expects the queue to clear in the next 24 hours.
- Webcast File Upload and Doc convert -- fix is in progress
- REACH captions -- temporary delays that should be cleared today

We will continue to monitor these issues with the vendor and provide updates as they are available.
Posted Sep 01, 2020 - 10:20 EDT
Update
Kaltura services should be operating as expected today, except:

· Transcoding time may be slower than normal due to backlog processing, though this should clear up today;
· Document conversion which should resolve today;
· Some upload issues which should resolve today.

We will continue to monitor these issues with the vendor and provide updates as they are available.
Posted Aug 31, 2020 - 16:05 EDT
Monitoring
The vendor has implemented a fix for the reported issues. We are continuing to monitor the issues with the vendor.
Posted Aug 27, 2020 - 16:46 EDT
Investigating
The vendor is currently investigating an issue with Kaltura. We will provide updates as we learn more.
Posted Aug 27, 2020 - 16:36 EDT
Monitoring
The vendor has indicated that the issue is resolved. We will continue to monitor the issue and provide updates as needed.
Posted Aug 27, 2020 - 15:04 EDT
Investigating
Kaltura uploads may be experiencing intermittent issues. The vendor is investigating this issue.
Posted Aug 27, 2020 - 14:52 EDT
Update
There may have been some overnight recurrences of issues related to viewing Kaltura content. The vendor is continuing to monitor the errors.
Posted Aug 27, 2020 - 08:40 EDT
Monitoring
The vendor has fixed the issues that were impacting Kaltura videos, and videos should be playing normally at this time. We are continuing to monitor this issue with the vendor. If you do continue to have problems with Kaltura, please contact help@gsu.edu.
Posted Aug 26, 2020 - 14:10 EDT
Investigating
Kaltura is currently experiencing problems, and videos may not be available at this time. We are working with the vendor to identify and resolve the issue.
Posted Aug 26, 2020 - 11:10 EDT
This incident affected: Kaltura.