Attempts of Security Breach
Incident Report for AudienceView Campus
Resolved
This incident has been resolved.
Posted Apr 12, 2023 - 07:33 EDT
Update
SFTP set up is completed. Please upload any new files today for reflecting the user data on your side.

The issue reported last week with emails being filtered to spam/junk folders has been cleared. Emails for password resets and order confirmations from no-reply@audienceview.com should now appear in your customers primary inbox.

If Print server icon stays red
i. Go to your control panel and uninstall the print server application
ii. Go to this article (https://audienceview2.my.site.com/Campus/s/article/Installing-a-Printer?language=en_US) and click on "DOWNLOAD THE EXE FILE"
iii. Once installed connect to the print server application and follow the steps in the article.
Posted Mar 20, 2023 - 16:41 EDT
Update
We have had clients reporting 2 issues with printing and connecting to the print server.

1. Print server icon stays red
2. Print server is green but the QR codes don't print.

For issue 1. You will have to uninstall and reinstall the print server application:
i. Go to your control panel and uninstall the print server application
ii. Go to this article (https://audienceview2.my.site.com/Campus/s/article/Installing-a-Printer?language=en_US) and click on "DOWNLOAD THE EXE FILE"
iii. Once installed connect to the print server application and follow the steps in the article.

For issue 2 we are still investigating. We still do suggest you download the updated print server application.  

We have also been receiving reports of emails for password resets and confirmation emails not going to customers inboxes. It looks as though the emails are filtering to spam so please advise your customers to check their spam/junk folder. We are investigating further to resolve this in the meantime.
Posted Mar 16, 2023 - 17:22 EDT
Monitoring
We are updating the status of this incident to Monitoring, for the majority of clients fully functionality is restored. There are some outstanding issues that we continue to investigate, please see the list below:

1. We continue to investigate issues on a case by case basis with regards to SSO, Payment Gateways, Vanity URLs.
2. We continue to investigate the printing issues that have been reported by clients.
3. SFTP connection is still working to be restored we expect to be able to provide an update at the end of the week.
4. The cloudflare enable cookies error messages continue to appear for some clients. If you continue to see these errors on your site please log a case with support and provide the ray ID and IP that is listed.
5. Some clients have been reporting slowness and inability to pull financial reports, we suggest attempting to pull a smaller report/date range. If you continue to have issues please submit a support case
Posted Mar 15, 2023 - 14:23 EDT
Update
We appreciate your patience as we work through the various issues that have been reported.

SMS mobile tickets not sending – Resolved

Oops error in confirmation page – Resolved

E-tickets not downloading - Resolved

Cloudflare – investigating – Please provide us with the Ray ID and the IP (at the bottom of the error page) and send that to support. We will use this to help troubleshoot.

Vanity URLs – in progress - On Friday we sent updates to all clients who have a custom Vanity URL and need to make updates on their end. Once the updates are made on your end, we need to make a final validation. Please communicate with support if you have any questions, haven’t received any information, or are still unable to use the Vanity URL.

SSO – If you continue to experience issues with SSO please reach out to support and we will investigate on a case-by-case basis.

Some of the issues marked as resolved were updated as recently as 30 minutes ago. Please check your site to confirm the fixes are working and if you are still encountering problems reach out to support for further investigation.

If you have a case with support open and the issue is now resolved, we ask that you set the status to ‘Closed by client’ or email us to let us know. This will help us to focus on cases that still need our attention.
Posted Mar 13, 2023 - 11:45 EDT
Update
The team is aware of notification SMS not delivered. As an alternative please use the Eticket option until the issue is resolved.

We have received confirmation from our team and clients have confirmed that SSO is now functioning as expected. If you are still having issues with SSO please log a case.

Payment gateway updates continue to be made as we update the credentials on our end. If you haven’t yet, please update your gateway to whitelist our IP address and send us your updated API Key. If you are running into issues with payment processing, please submit a P1 case and we will address those concerns one by one.

We have sent information to all clients who own their own DNS record so that you can make the changes necessary to get your Vanity URL working. The Vanity URLs that AudienceView manages are now functional.

Clients continue to encounter a ‘cookie error’ when editing the site. Please clear your cache and cookies and try to reload the page. If you can capture the ‘Ray ID’ at the bottom of the error page and submit a case with those details, we can investigate further.

We have received reports from some clients that they are receiving an oops error message on the order confirmation emails. If you are seeing this, the download e-tickets and download mobile tickets buttons should still be functioning. We will investigate the oops error message early next week. If you are unable to download e-tickets please submit a case and provide details (operating system version, browser) so we can investigate.
Posted Mar 12, 2023 - 14:35 EDT
Update
We have received confirmation from our team and clients have confirmed that SSO is now functioning as expected. If you are still having issues with SSO please log a case.

Payment gateway updates continue to be made as we update the credentials on our end. If you haven’t yet, please update your gateway to whitelist our IP address and send us your updated API Key. If you are running into issues with payment processing, please submit a P1 case and we will address those concerns one by one.

We have sent information to all clients who own their own DNS record so that you can make the changes necessary to get your Vanity URL working. The Vanity URLs that AudienceView manages are now functional.

Clients continue to encounter a ‘cookie error’ when editing the site. Please clear your cache and cookies and try to reload the page. If you can capture the ‘Ray ID’ at the bottom of the error page and submit a case with those details, we can investigate further.

We have received reports from some clients that they are receiving an oops error message on the order confirmation emails. If you are seeing this, the download e-tickets and download mobile tickets buttons should still be functioning. We will investigate the oops error message early next week. If you are unable to download e-tickets please submit a case and provide details (operating system version, browser) so we can investigate.
Posted Mar 11, 2023 - 12:37 EST
Update
We have sent information to all clients who own their own DNS record so that you can make the changes necessary to get your Vanity URL working.

We have resources available over the weekend to work on your support cases if your site is inaccessible, and if your payments or SSO are not working.

If you are running into issues with orders, etickets, and printing we ask that you review this resource for steps on how to handle an event without these features.
https://audienceview2.my.site.com/Campus/s/article/Tech-Outage-Tips?language=en_US

We are prioritizing site access and payment processing at this time due to the volume of cases and will do our best to get to everything. We appreciate all your patience.
Posted Mar 10, 2023 - 17:33 EST
Update
We have fixed the Vanity URLs issue for all clients where we own the DNS record.

For all clients who own their own DNS record, we are currently sending this information to you.

We have resolved the “Please enable Cookies and reload the page.” error when editing for some clients. If you are still having issues, please open a support case and provide the RAY ID to us to expedite resolution.

We continue to investigate SSO and payment gateway issues on a case-by-case basis. If you are still running into issues please reach out to support.
Posted Mar 10, 2023 - 14:27 EST
Update
For clients who have reported SSO issues, we made changes last night to resolve the issue. Please verify that this is working today and let us know if this is resolved by closing your support case directly or follow up to confirm things are working. If this isn’t resolved as of this morning let us know and we can investigate further.

Some clients are reporting getting into a waiting room and there are some errors appearing. We are working on resolving this today.

For clients with Vanity URLs, where we own the DNS record, we are working to resolve this today. For clients who own their own DNS record, we will be sending you the necessary information to update your DNS record today. You will need to make the updates on your side. Base URLs are now working for all clients.

Some clients have reported a “Please enable Cookies and reload the page.” error when editing. Our Open Office session at noon today will be dedicated to this issue. If you are impacted, we invite you to attend the Open Office session where we will have our security experts available.

We continue to work on scheduled reports, and SFTP connections for uploads and downloads. At this time, a resolution is expected next week at the earliest.

If you have encountered any issues not mentioned here, please reach out to support.
Posted Mar 10, 2023 - 09:26 EST
Update
We have resolved the printing issue.  

Some clients have reported a “Please enable Cookies and reload the page.” error when editing. We are investigating this.  

Some clients are reporting getting into a waiting room and there are some errors appearing. If you are experiencing this please report to support and we can assist. 

For clients with Vanity URLs we are still working on granting access to these sites. We expect by end of day today you should have access to your base URL site e.g. schoolname.universitytickets.com . We expect to be ready with the Vanity URL redirects tomorrow. 

For clients who have been reporting SSO issues, we continue working to resolve this as soon as possible. 

We continue to work on scheduled reports, and SFTP connections for uploads and downloads.  

If you have encountered any issues not mentioned here please reach out to support.
Posted Mar 09, 2023 - 18:07 EST
Update
Currently we have 100% of our client’s systems fully back online. As a reminder, the following features are not available. We continue to work on these and will provide an update as soon as possible.

1. Printing
2. Scheduled reports
3. SFTP connections for uploads and downloads
4. Vanity URLs

IMPORTANT: When you first get access to your system, we recommend that you run a test sale on WEB and POS sale to confirm the payment gateway connection is functioning.

For steps on how to disable online sales please review this article.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US#web-sales

As a reminder, all accounts will be asked to reset their password upon first login. That process is outlined here.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US#login

If you run into any issues, please contact support.
Posted Mar 09, 2023 - 14:57 EST
Update
Currently we have 85% of our client’s systems fully back online. We appreciate your patience as we work on getting everyone up.

We have resolved the SSO issues. We have also resolved errors when editing events. If you are still running into issues, please contact support.

Note that due to increased security, there will be occasions when you or your ticket buyer might encounter a challenge, such as CAPTCHA.

As a reminder, we are hosting an open office hours session today at 3pm ET. You can find the link for that session here.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US
Posted Mar 09, 2023 - 14:28 EST
Update
Currently we have 60% of our client’s systems fully back online.

We appreciate your patience as we work on getting everyone up. We are currently investigating issues with access through SSO. If you are receiving an error on SSO please reach out to support and we will make the necessary updates on our end.
Posted Mar 09, 2023 - 13:12 EST
Update
Currently we have 30% of our client’s systems fully back online.

We appreciate your patience as we work on getting everyone up.
Posted Mar 09, 2023 - 11:57 EST
Update
Currently we have 10% of our client’s systems fully back online.

We appreciate your patience as we work on getting everyone up.
Posted Mar 09, 2023 - 10:59 EST
Update
We are happy to announce that we are starting to bring the Campus system back online shortly. Please note that we expect this process to take 4-5 hours. We will continue to update you on an hourly basis here.

Please note that we are expecting a temporary 503 error on your site during this time. This is expected, so please do not be concerned. Once your site is back online, this message will disappear.

As a reminder, the following features will not be available to you at the moment:
1. Printing
2. Scheduled reports
3. SFTP connections for uploads and downloads
4. Vanity URLs

If you haven’t requested that we turn off your web sales by now, then you can expect sales to be on when your site goes live. Please refresh your website through the day, this will be the quickest way to see when you have access to your site.

For steps on how to disable online sales please review this article.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US#web-sales

As a reminder, all accounts will be asked to reset their password upon first login. That process is outlined here.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US#login

IMPORTANT: When you first get access to your system, we recommend that you run a test sale on WEB and POS sale to confirm the payment gateway connection is functioning.

If you run into any issues, please contact support. If you still need to update your API key then please review the steps in this article.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US

If you have any questions and would like to discuss in person, we are opening an open office hours session today at 3pm ET. You can find the link for that session here.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US

We appreciate your patience as we work to get all the sites up and functional today.
Posted Mar 09, 2023 - 10:13 EST
Update
Thank you for your patience as we address the security incident regarding our Campus product. We are pleased to report that in collaboration with our third-party cybersecurity experts, we have contained the incident and will begin bringing your system back online starting tomorrow, March 9 between 11am –3pm ET.

We have also implemented additional safeguards and taken measures to help prevent further unauthorized access, including:

1. implementing a web application firewall (Cloudflare) and enabling Microsoft Defender for Endpoint on all servers, in addition to our existing
Carbon Black deployments;
2. conducting a penetration test in relation to the Campus product;
3. resetting credentials out of caution;
4. implementing defensive controls on all indicators identified during the incident; and
5. implementing best-practice security controls for further strengthening our security posture.

To get you back up and running quickly, some features will not be available immediately. We will work to bring these features back as quickly as possible once all clients are back online.

The features that won’t be available to you immediately are:

1. Scheduled reports
2. SFTP connections for uploads and downloads
3. Vanity URLs

For the most up-to-date information tomorrow as we start bringing your system back online, please subscribe to this AudienceView Campus Status Page. As a reminder, please continue to check this resource. We continue to host Open Office Hour sessions through to the end of this week, to help answer your questions. Details about these sessions are posted here.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US

We appreciate your patience and look forward to bringing your system back online.
Posted Mar 08, 2023 - 17:22 EST
Update
We are pleased to announce that we are on track to have the Campus product accessible to you this week. Our team is working tirelessly to complete the fortification of our Campus infrastructure and product, in consultation with Mandiant. These changes increase the security and protection of your system. We are running testing now to ensure the sites will come back fully functional. 

Please review this resource in our learning portal with information on what you need to do to prepare for your site coming back online. There are some action items to be sure that payment processing can be completed.
https://audienceview2.my.site.com/Campus/s/campus-system?language=en_US

We will also be offering open office hours at 12pm ET today, the link to register is in the learning portal article.
Posted Mar 07, 2023 - 09:03 EST
Update
We would like to update you on the progress we've made in restoring access to your sites after the recent security incident. Our team has been working around the clock all weekend, in close collaboration with Mandiant, to ensure that your data is safe and secure. 

Our current estimate is that we will be able to bring your sites back online by mid-week. We understand that this has put you in a difficult position with your customers and we appreciate your patience as we work towards a safe solution. 

Later today, we will be sending out more information on how to prepare for turning your sites back online.
Posted Mar 06, 2023 - 11:32 EST
Update
Our team of cyber security experts from Mandiant is currently conducting a thorough forensic investigation into the breach. In the meantime, we are taking necessary steps to fortify the Campus infrastructure before reopening.  

We understand how difficult this situation is and thank you for your continued patience while we work to safeguard your system.     

If you have any further questions, please contact campus.support@audienceview.com and we’ll triage you to the best resource.
Posted Mar 03, 2023 - 11:06 EST
Update
We wanted to provide a brief update on the status of the AudienceView Campus product. Our cyber security consultants from Mandiant continue their investigation and analysis of our systems to ensure the security threat has been removed and we are secure against any future attempts. This remains our top priority before opening up access to your system.   

As a reminder, we can assist with your event management:   
1. We have pulled attendee lists for your events in the next two weeks. Contact Support at campus.support@audienceview.com to request
this data to help you manage your events and your customers offline.  

2. We have compiled a guide to running offline events you may find helpful.  
https://audienceview2.my.site.com/Campus/s/article/Tech-Outage-Tips?language=en_US

We are still recommending that you plan on managing your events offline for the next 4-6 days without access to the system. We understand how difficult this situation is and thank you for your continued patience while we work to safe guard your system.   

If you have any further questions, please contact campus.support@audienceview.com and we’ll triage you to the best resource. For the most up-to-date information, please subscribe to the AudienceView Campus Status Page.
Posted Mar 01, 2023 - 15:11 EST
Update
Our staff is working hard to get through all of your questions. At this time the most up to date information that we have on this incident is being posted on the campus status page. In an effort to streamline communications we ask that you hold back questions about when the sites will come online, as this information will be communicated to everyone via the status page. 

We are prioritizing requests from clients who have events happening today. If you need reservation lists for your upcoming events please proceed with submitting a case to support or emailing campus.support@audienceview.com

If you have submitted a case we will get to you, submitting a second case could slow down the process and the time it takes our support staff to respond to you. 

We appreciate your patience as we work through these requests.
Posted Feb 28, 2023 - 15:38 EST
Update
As our investigation of this security threat continues, we’ve been informed that your Campus system will need to be kept offline for a longer period of time than originally anticipated. We recommend you plan on managing your events offline for the next 5-7 days without access to the system.  

We understand that this negatively impacts your business and we sincerely apologize. We must, however, ensure that all system threats are mitigated and the system maintains resiliency from future unauthorized attacks. This remains our top priority. 

Meanwhile, we will assist with your event management:  
1. We have pulled attendee lists for your events in the next two weeks. Contact Support at campus.support@audienceview.com to request this data to help you manage your events and your customers offline.  
2. We have compiled a guide to running offline events you may find helpful.  
https://audienceview2.my.site.com/Campus/s/article/Tech-Outage-Tips?language=en_US

We will provide further updates to the status page as we have more information.
Posted Feb 28, 2023 - 12:18 EST
Update
We continue to investigate this and as a safety precaution we will keep sites offline until further notice. We have reached out to all clients who have events this evening and provided reservation list reports to help with their evenings events.

If you have events upcoming in the following days and would like ticket reservation data for your reports please contact campus.support@audienceview.com and we will be happy to assist. 

We will have another update for you in the morning.
Posted Feb 27, 2023 - 19:00 EST
Update
We continue to investigate the situation and your sites remain offline.

We understand this has an impact on your business and you have upcoming events that need to be managed. We are currently assessing the best way to assist your event management needs and will be following up to any clients that have events in the near-term with a separate set of instructions.

We will provide regular updates and will do everything we can to assist you during this challenging time.
Posted Feb 27, 2023 - 15:14 EST
Update
We have temporarily turned off access to your Campus instance to reduce the risk of a perceived security threat. We will provide additional updates as soon as possible through this page.
Posted Feb 27, 2023 - 12:57 EST
Investigating
We have seen evidence of further attempts to breach our security of the Campus product. In an abundance of caution, we will be turning online sales off until we have completed our investigation. Additional updates will be provided though this page.
Posted Feb 27, 2023 - 11:28 EST
This incident affected: Admin Menu, Email/SMS Notification, Gift Certificates, Payment Processing, Printing, Reporting, Sales - Web or POS, Single Sign On, and Ticketing Site.