All
Systems Go (except any listed below)....
.........................................................................................................................................
We experienced a Service Outage which started on the 10th April 2008 at 09:50 and was resolved by 1400 10/04/08.
For your information a summary of this outage is below:
| Category: | Service Outage (ADSL) |
| Service Element: | Broadband Aggregation Network |
| Time the Outage occurred: | on the 10th April 2008 at 09:50 and was ongoing until resolution at 14:00 |
| Description: | We noticed a session drop across multiple devices in multiple locations on the aDSL network. Due to the distribution of this it is our belief that this is currently a BT issue. All LTS and LNS devices are functioning without issue and serving sessions. Our wholsale suppliers NOC is currently liaising with BT to pin point the issue. We will update customers as soon as we have a further update. |
We apologise for any inconvenience this outage may have caused.
For your information a summary of the update is below:
| Notes: | This ticket is now closed. The issue was successfully resolved at 1400 10/4/08 by liasing with our Wholesaler and BT. |
We apologise for any inconvenience this issue may have caused.
Regards.
Customer Services.
.........................................................................................................................................
We are currently experiencing a Service Outage which started on the 29th March 2008 at 12:33 and is still ongoing.
For your information a summary of this outage is below:
| Category: | Service Outage |
| Service Element: | Broadband Aggregation Network |
| Time the Outage occurred: | on the 29th March 2008 at 12:33 and is still ongoing |
| Description: | Dear Customer, Green ISP has experienced a Cisco LTS hardware failure in Telehouse North, this has affected around 400 L2TP sessions. Due to the resilient nature of the network these sessions immediately returned via alternative routes thus customers will have only experienced a very very brief interuption of service. We apologise for this inconvenience. Traffic will continue to be re-routed via the resilient paths whilst the hardware issue is investigated and if necessary the device replaced. |
We apologise for any inconvenience this outage may cause and will be issuing further updates until resolution.
For your information a summary of the update is below:
| Notes: | This ticket is now closed. The repair was successfully completed without any major interuption to service. |
We apologise for any inconvenience this issue may have caused.
Regards.
Customer Services.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PLANNED ESSENTIAL WORKS 8/3/08 at 00:01 until 0600:
As a valued customer, Green ISP is committed to keeping you informed about any changes in the status of your service with us.
This is an advance notification of Planned Engineering Works.
For your information a summary of this work is below:
| Category: | Planned Engineering Works |
| Service Element: | Broadband Aggregation Network |
| Maintenance Window: | The work will start on 8th March 2008 at 00:00 and is scheduled to end at 06:00 |
| Description: | Engineers will be performing routine maintenance to the Broadband Aggregation routers and switches to further aid in the expansion of the network. Whilst this work is not expected to impact service, customers should deem the service at slight risk whilst this maintenance is taking place. |
For your information a summary of the update is below:
| Notes: | This ticket is now closed. The maintenance was successfully completed this morning without any interuption to service. |
We apologise for any inconvenience this essential maintenance may have caused.
Regards.
Customer Services.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PLANNED ESSENTIAL WORKS: (The work will start on 5th March 2008 at 06:00 and is scheduled to end at 06:05):
As a valued customer, Green ISP is committed to keeping you informed about any changes in the status of your service with us.
This is an advance notification of Planned Engineering Works.
For your information a summary of this work is below:
| Category: | Planned Engineering Works |
| Service Element: | Broadband Aggregation Network |
| Maintenance Window: | The work will start on 4th March 2008 at 22:00 and is scheduled to end on 5th March 2008 at 06:00 |
| Description: | Due to the way in which BT perform their loading of circuits across the IP Stream BT centrals the NOC managers have noted a steady increase and session inbalance with the capacity terminated within Telehouse North. As a result of this it is necessary for them to perform session rebalancing to ensure continued and sustainable high speed and low latency services for these end users. |
| Service Outage Expected: | Yes |
| Expected Outage Duration: | 5 Mins |
| Service Impact Description: | Engineers will perform this maintenance work, which will involve a brief session disconnected and immediate reconnection for a number of end users terminated within this location when the network traffic is at its lowest. We apologise for the inconvenience this brief interruption in service will cause whilst we work to continue to deliver premium customer experience. |
For your information a summary of the update is below:
| Notes: | This ticket is now closed. The maintenance was successfully completed this morning without any interuption to service. |
We apologise for any inconvenience this essential maintenance may have caused.
Regards.
Customer Services.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PLANNED ESSENTIAL WORKS: (The work will start on 23th January 2008 at 00:00 and is scheduled to end at 06:00):
As a valued customer, Green ISP is committed to keeping you informed about any changes in the status of your service with us.
This is an advance notification of Planned Engineering Works.
For your information a summary of this work is below:
| Category: | Planned Engineering Works |
| Service Element: | Broadband Aggregation Network |
| Maintenance Window: | The work will start on 23th January 2008 at 00:00 and is scheduled to end at 06:00 |
| Description: | Engineers will be performing upgrades to our core switches in our Redbus Meridian PoP on Thursday 23th January, 12am - 6am. These upgrades will further bolster capacity and resilience at this site. |
| Service Outage Expected: | Yes |
| Expected Outage Duration: | 1 hour |
| Service Impact Description: | Whilst these upgrades are taking place non resilient L2TP customers may experience a brief drop in sessions, transit connectivity to the public internet may also be sporadic during these upgrades. The upgrades will take no longer than 1 hour to complete. |
For your information a summary of the update is below:
| Notes: | This ticket is now closed. The maintenance was successfully completed this morning without interuption to service. |
We apologise for any inconvenience this essential maintenance may have caused.
Regards.
Customer Services
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PLANNED ESSENTIAL WORKS: (21st January 2008 at 00:00 and is scheduled to end at 19:00):
As a valued customer, Green ISP is committed to keeping you informed about any changes in the status of your service with us.
This is an advance notification of Planned Engineering Works.
For your information a summary of this work is below:
| Category: | Planned Engineering Works |
| Service Element: | Broadband Aggregation Network |
| Maintenance Window: | The work will start on 21st January 2008 at 00:00 and is scheduled to end at 06:00 |
| Description: | Engineers will be performing upgrades to the core fibre switches in our Redbus Meridian PoP on Saturday 21st, 12am - 6am. These upgrades will further bolster capacity and resilience at this super PoP node. |
| Service Outage Expected: | Yes |
| Expected Outage Duration: | 1 hour |
| Service Impact Description: | Whilst these upgrades are taking place non resilient L2TP customers may experience a brief drop in sessions whilst this work is taking place. The upgrades will take no longer than 1 hour to complete. |
For your information a summary of the update is below:
| Notes: | This ticket is now closed. The maintenance was successfully completed this morning without interuption to service. |
We apologise for any inconvenience this essential maintenance may have caused.
Customer Services
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PLANNED ESSENTIAL WORKS: (11th January 2008 at 19:00 and is scheduled to end on 12th January 2008 at 19:00):
As a valued customer, Green ISP is committed to keeping you informed about any changes in the status of your service with us.
This is an advance notification of Planned Engineering Works.
For your information a summary of this work is below
| Category: | Planned Engineering Works |
| Service Element: | Service Delivery Network |
| Maintenance Window: | The work will start on 11th January 2008 at 19:00 and is scheduled to end on 12th January 2008 at 19:00 |
| Description: | We have received a maintenance notification from one of our fibre operators in London Docklands detailing the requirement to repair a number of fibres that have been erroded since installation 3 years ago. This maintenance will affect the fibre link between Harbour Exchange and Telehouse North however will not be service affecting as traffic will re-route round the resilient ring. Services should be deemed at risk whilst this maintenance is taking place as resilience between these points will be lost. |
| Service Outage Expected: | AT RISK |
| Expected Outage Duration: | NONE |
| Service Impact Description: | Subscribers should not experience any downtime however there is a risk due to resilience between these points being lost until work complete. |
For your information a summary of the update is below:
| Notes: | This ticket is now closed. The maintenance was successfully completed this morning without interuption to service. |
We apologise for any inconvenience this essential maintenance may have caused.
Customer Services.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
23/11/07 - 12:37
ADSL SERVICE OUTAGE (now resolved 13:05):
For your information a summary of this outage is below:
| Category: | Service Outage |
| Service Element: | Broadband Network |
| Time the Outage occurred: | on the 23rd November 2007 at 12:37 and this issue has now been resolved 13:05 |
| Description: | Engineers are currently investigating a number of subscriber disconnects from the network. Engineers are also now working to identify the issue and will update customers accordingly. |
For your information a summary of the update is below:
| Notes: | The majority of sessions have now restored, if you are still experiencing issues please power cycle your router to rejoin the network. 13:30 Most customer ADSL sessions are now back online. |
We apologise for any inconvenience this outage may have caused.Regards.Green ISP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PLANNED ESSENTIAL WORKS BY BT (Starting 15/10/07 - ending 19th October 07):
As a valued customer, Green ISP is committed to keeping you informed about any changes in the status of your service with us.
This is an advance notification of Planned Engineering Works.
For your information a summary of this work is below
| Category: | Planned Engineering Works |
| Service Element: | Broadband Aggregation Network |
| Maintenance Window: | The work will take palce on the 15th, 16th, 17th, 18th and 19th October 2007 each day at 00:01 and is scheduled to end at 06:00 |
| Description: | BT will be carrying out engineering work to upgrade the DSLAM in various exchanges across the country that will affect the Broadband Aggregation Network. |
| Service Outage Expected: | Yes |
| Expected Outage Duration: | 1 hour |
| Service Impact Description: | Subscribers may experience an outage of up to an hour while the upgrade work is being done. Once the work has been completed subscriber sessions will re-connect. |
We apologise for any inconvenience this essential maintenance may cause and will be issuing further updates until resolution.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Incident Report:
Date: 25th Sep 2007
Incident Date: 25th Sep 2007Introduction:
This document provides an explanation of the issues that resulted in customers
being disconnected from the network on Sep 25th 2007.Details:
At 8am this morning our Network Operations Team was notified of an issue that
was affecting the performance of our Service Delivery Network (SDN) in the
Meridian Gate datacentre. This resulted in a number of customers experiencing
loss of service with their IP sessions being terminated and unable to immediately
re-establish.After investigation our NOC concluded the issue was relating to the SDN cluster
operating in Meridian. The VRRP (failover) software operating this cluster had
experienced an issue where none of the servers were taking primarily ownership
of the traffic, thus with all servers operating as secondary slaves and awaiting
instruction from the primary traffic was subsequently dropped. Once this issue
was resolved sessions began to re-establish. Our Network Team have utilised this software
since 2001 and have never to date experienced this issue.
Within 20 minutes of the problem with the SDN routers being resolved, the
majority of customers had reconnected. The remaining customers had
reconnected within a further 30 minutes due to the customers being brought back
in a controlled fashion to prevent any further degredation to the service.Some Customers will have experienced rejected login attempts due to the increased
volume of authentication attempts.Resolution:
The Network Analyst team have recently been deploying a number of network upgrades that will
allow for this section of BT capacity to immediately terminate in a backup
location. They have now advanced the upgrade schedule to imminently deploy
these changes. Issues such as this are incredibly rare however Green ISP will
continue to ensure that ongoing resource is dedicated to ensuring the network
remains resilient and reliable, via our Systems and Network analysts utilised.Sorry for the inconvenience caused.Regards.Green ISP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PLANNED UPGRADE (Starting 15/9/07 - for about a 2 week window):
As part of this upgrade customers will benefit from decreased latency and faster speeds.
The engineers will also be taking this opportunity to upgrade customers products to DSL Max, whilst initially service will be "speed limited to the customers contracted speeds" customers will be able to upgrade to faster max speeds in the future with ease.
No username, password or IP address changes will be required as part of this upgrade.
This upgrade will involve some customers connection being forceably disconnected within our maintenance window just once during this 2 week window, the circuit will then immediately reconnect on the alternative central pipes and service will be back to normal.
Once customers are upgraded they will experience the 10 day retraining window for customers on Up to 8Mb ADSL Max services, this can involve a limited number disconnections for up to 10 days. If any customer is experiencing issues following this upgrade please contact our helpdesk in the usual way or give us a call 0870 7201690.
Support eTicket system
http://www.greenisp.org.uk/eticket_support/
DETAILS OF SCHEDULE:
For your information a summary of this work is below:
| Category: | Planned Engineering Works |
| Service Element: | Broadband Aggregation Network |
| Maintenance Window: | The work will start on 15th September 2007 at 01:00 and is scheduled to end at 01:30 |
| Description: | Engineering work is required on the Broadband Aggregation Routers to enable hardware upgrades to be completed. |
| Service Outage Expected: | Yes |
| Expected Outage Duration: | 5 minutes |
| Service Impact Description: | Subscribers will usernames ending with @surfdsluk, @surfdsluk.net, and @surfdsluk.com will be disconnected. Once the work has completed, subscribers will be able to reconnect their sessions. |
| Category: | Planned Engineering Works |
| Service Element: | Broadband Aggregation Network |
| Maintenance Window: | The work will start on 22nd September 2007 at 00:00 and is scheduled to end at 06:00 |
| Description: | Engineering work is required on the Broadband Aggregation Network, essential relocation of core network equipment will be carried out. |
| Service Outage Expected: | Yes |
| Expected Outage Duration: | 1 hour |
| Service Impact Description: | There will be an outage of up to a hour during the maintenance window. This will affect all xDSL subscriber's sessions. Once the work has completed, subscribers will be able to reconnect their sessions. |
We apologise for any inconvenience this essential maintenance may cause and will be issuing further updates until resolution.
Thanks for your undersanding.
Regards.
Green ISP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
26/08/07 - 0800
PoP mail service intermittant problems (not all customers are effected):
pop3.greenisp.org (incoming mail server) is experiencing performance issues this morning, customers may not get a response from the server for retreiving new email, we have engineers investigating this and hope to have this resolved very soon.
Pleease accept our apologies for the inconvenience cuased.
Regards.
UPDATE/RESOLUTION 26/8/07 - 1200
This issue has now been resolved.
However customers are advised that if they are still experiencing problems, please shut down your email client (i.e. Outlook) or the WebMail system, restart it and try again.
Sorry for the inconvenience this may have caused.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
15/8/07 - 1800
Web Mail and Control panel access issue (cp.greenisp.org domain related):
At 1800 today 15/8/07 there was an issue with our greenisp.org domain name, this is used to deliver email and Customer Control panel access via cp.greenisp.org and mail.greenisp.org.
Unfortunately there was an unavoidable DNS (Domain Name Server) issue with the .org registry in the US which lost the DNS record for the domain temporarily. On visiting greenisp.org customers would have been directed to the registries "default holding page" which would have shown a list of Internet resources, (Internet access was not effected in any way).
We are now waiting for the fix which has been effected to replicate around the worlds DNS systems.
Sorry for the inconvenience this may have caused.
UPDATE/RESOLUTION 16/8/07 - 0025
This issue has now been resolved.
However customers are advised that if they are still experiencing problems logging into the Green ISP control panel and accessing their WebMail they should Clear all "cache" from their Web Browser then restart the browser and try again.
Again, Sorry for the inconvenience this may have caused.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
23/7/07
This is an advance notification of planned Emergency Maintenance.
For your information a summary of this work is below:
Category: |
Emergency Maintenance |
Service Element: |
Broadband Aggregation Network |
Maintenance Window: |
The work will start on 25th July 2007 at 01:30 and is scheduled to end at 02:30 |
Description: |
Engineering work is required on the Broadband Aggregation Network to facilitate further upgrades. This maintenance will not be service affecting however services are deemed at risk whilst this maintenance takes place. |
11/07/07
This is an advance notification of planned Emergency Maintenance.
For your information a summary of this work is below:
Category: |
Emergency Maintenance |
Service Element: |
Broadband Aggregation Network |
Maintenance Window: |
The work will start on 12th July 2007 at 02:00 and is scheduled to end at 03:00 |
Description: |
Engineering work is required on the Broadband Aggregation Network to facilitate further upgrades. This maintenance will not be service affecting however services are deemed at risk whilst this maintenance takes place. |
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
16/05/07 @ 1800
Network equipment issue effecting some customers
This is an explanation of the issues that resulted in customers
being disconnected from the network on May 16th 2007.
Details:
Engineers were on-site performing non service affecting maintenance
during May 16th 2007, at 5pm the engineers decommissioned a backup core
router to be upgraded following the upgrade of the primary router during maintenance a few weeks ago.
At 5.15pm whilst the upgrade was being performed engineers noticed a number
of disk errors on the core router, whilst being investigated unfortunately the core router crashed terminating user sessions. These sessions would usually be immediately picked
up and continued by the secondary router however this had yet to fully boot and
be re-commissioned after its upgrade. This resulted in user sessions being fully
terminated and the customers being required to re-authenticate on the radius (authentication) server.
Whilst the primary and secondary routers were in full service by 5.30 - 6.00pm
customers may have experienced delays re-establishing service whilst the radius
attempted to re-authenticate the entire user base. The majority of customers re-
established sessions within the hour or so however a number of customers
experienced delays that resulted in their session not being fully established for up
to 3 hours.
Resolution for future.
Over the past months there has been radius enhancements that will
allow users to authenticate on services considerably faster than previously
experienced. There were plans to deploy these enhancements within the next
month.
This upgrade will now be brought forward and will be performed within the
next week.
Please accept our most sincere apologies for the issue and inconvenience you
have experienced. We always strive to ensure the network remains
consistently reliable, further enhancements will be performed to ensure that
these issues are not repeated in future.UPDATE/RESOLUTION - 17/5/07:
This issue has now been resolved, however customers are advised to re-boot all their ADSL and computer equipment to establish a clean session.
Please accept our most sincere apologies for the issue and inconvenience you
have experienced.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
5-6/4/07 @ 1600
We experienced a systems issue over the "bank holiday" weekend which may have effected some customers aDSL Broadband and Dial-up connections, this would have reported errors when trying to log in (authenticate) or the connection would be lost for a while.
Engineers were called out to resolve this issue which reoccurred a few hours later, then finally the issue was resolved on 6th April.
Not all customers were effected and the system was down on a couple of occasions for a couple of hours at a time.
NOTE:
If you are still experiencing performance issues please restart all equipment and try again (to establish a clean session).
The service may need time to settle back down after such a failure so please be patient over the next few days starting 7-8/4/07.
Thanks.
Sorry for any inconvenience this may have caused and thanks for your patience in this matter.
*This is the only major connectivity outage experienced since 10/7/2006.
UPDATE/RESOLUTION - 7/4/07:
We can now confirm the problem affecting some customers between 6.43pm and 10.23pm of the 5th April were due to ODF termination issues with the fibre provider within the Telehouse North PoP.
Systems engineers were despatched to site and reterminated via a backup VLAN restoring the service for customers.
Customers should not have been affected by this after 11.20pm on the 5th, however there may have been a lag period of about 24 hours (into the 6th of April) where some customers may have experienced some connection issues or minor disturbances.
ALL Systems back to normal on 6/4/07
Sorry again for the inconvenience this may have caused.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
1200 - 1/3/07
We are experiencing a Routing issue with web traffic external to our network, we are investigating the cause of this and will update shortly (this may not effect all customers and some more than others, depending on their physical location).
Sorry for any inconvenience this may cause.
UPDATE/RESOLUTION:
Resolved 1400 - 1/3/07
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
0900 - 01/08/06
We are currently experiencing a problem with our mail infrastructure storage network. This will affect POP3 and webmail services. Inbound mail is currently being queued and will be delivered once the issue has been resolved. Outgoing mail is not affected by this issue. The engineers are treating this as a high priority and are working to rectify the issue as soon as possible.
UPDATE - 2000 - UPDATE/RESOLUTION:
*This Issue is now resolved, engineers have worked to fix this problem.
This was also effecting some customers accessing their "Web Mail" at http://cp.greenisp.org
(the webmail menu is not appearing).
Sorry for any inconvenience caused.
(please keep trying to access your e-mail, as we may have a resolution soon)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
1300-1312 - 10/07/2006
Due to a routing issue on the DLS Network, some Broadband customers may have to try logging in several times before they get connected successfully, some customers may receive log-in (authentication errors), sorry for any inconvenience caused. The problem is currently being investigated and is being treated with the highest priority, however we do not currently have an ETA on resolution. We will let you know as soon as we have any further information.
UPDATE: 2100 -
10/07/2006 - UPDATE/RESOLUTION:
*Issue resolved all systems ok.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
1605 - 2nd March 2006
Due to a major system failure today most of our customers may experience problems connecting to the Internet.
We are working to resolve this issue as soon as possible:
UPDATE: 1700 - 2nd March 2006 - UPDATE/RESOLUTION:
*This fault/failure has now been resolved and customers should now be able to connect without issue, there may be a period of about 48 hours before full system performance is reached.
We would like to apologize for any inconvenience caused.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
0945 - 30th June 2005
ADSL BroadBand customers with usernames ending @surfdsl may be experiencing problems logging in to the service, there is a known issue with the service and we are working as quickly as possible to bring a solution to this.
It is estimated that this will be resolved by mid to late afternoon today.
We apologize for any inconvenience caused.
Please use the alternative 0845 local call rate access number.
Dial-up Number: 0845 0804999
Username: temp@greenisp
password: temp
update: 1500 - 1/7/05
This problem is due to a BT fault and is affecting a number of our @surfdsl username customers. The problem is currently being investigated and is being treated with the highest priority, however we do not currently have an ETA on resolution. We will let you know as soon as we have any further information.
We apologize for any inconvenience caused.
Please note that it is very rare and unusual for this type of incident to happen.
update: 1850 - 1/7/05
All systems are fully operational, customers should now reset their equipment (modems and routers) and be able to log in.
We again apologize for any inconvenience caused, and stress this is a very rare occurrence.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Green ISP Service.
eTicket Support System