Operations grimoire/Sentry - Revision history

Dereckson: /* Relay */

2024-01-13T17:14:07Z

Relay

← Older revision		Revision as of 17:14, 13 January 2024
Line 106:		Line 106:
	</syntaxhighlight>		</syntaxhighlight>

	As a workaround, put the sentry_web IP (<code>docker inspect sentry_web \| grep 172</code> to get it) in <~~syntaxhighlight~~>/srv/relay/sentry_relay/config.yml</code> file under <code>relay.upstream</code> key.		As a workaround, put the sentry_web IP (<code>docker inspect sentry_web \| grep 172</code> to get it) in <code>/srv/relay/sentry_relay/config.yml</code> file under <code>relay.upstream</code> key.

	If you've this message in the log, the sentry_web container is down or more probably its IP has changed and you also need to update it:		If you've this message in the log, the sentry_web container is down or more probably its IP has changed and you also need to update it:

Dereckson: Don't deploy OpenAI integration

2023-05-15T15:46:23Z

Don't deploy OpenAI integration

← Older revision		Revision as of 15:46, 15 May 2023
Line 81:		Line 81:
	\| [https://devcentral.nasqueron.org/D2881 D2881] \|\| [https://github.com/getsentry/self-hosted/commit/06662c6cdc8573f81b6a128c7bdb815efd5040ed 06662c6cdc85] \|\| 23.2.0		\| [https://devcentral.nasqueron.org/D2881 D2881] \|\| [https://github.com/getsentry/self-hosted/commit/06662c6cdc8573f81b6a128c7bdb815efd5040ed 06662c6cdc85] \|\| 23.2.0
	\|}		\|}

			== Non-features ==

			The following features can't currently be deployed:

			* https://github.com/getsentry/self-hosted/pull/2115/files - https://docs.sentry.io/product/issues/issue-details/ai-suggested-solution/ - lack of relevance of the solutions offered, privacy policy difficult to align, can't justify extra cost for the API subscription

	== Troubleshoot ==		== Troubleshoot ==
Line 86:		Line 92:
	=== Resources ===		=== Resources ===

	A ~~trouble shoot~~ page exists in Sentry dev documentation, we followed successfully the out-of-sync Kafka procedure for example:		A troubleshoot page exists in Sentry dev documentation, we followed successfully the out-of-sync Kafka procedure for example:
	https://develop.sentry.dev/self-hosted/troubleshooting/		https://develop.sentry.dev/self-hosted/troubleshooting/

Dereckson: /* Kafka */

2023-04-25T00:55:40Z

Kafka

← Older revision		Revision as of 00:55, 25 April 2023
Line 120:		Line 120:
	=== Kafka ===		=== Kafka ===

	After a memory issue on docker-002 and a hard reboot, the ~~entry_post_process_forwarder_transactions~~ and sentry_post_process_forwarder_errors containers exited on start with the following log message:		After a memory issue on docker-002 and a hard reboot, the sentry_post_process_forwarder_transactions and sentry_post_process_forwarder_errors containers exited on start with the following log message:

	arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}		arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}

Dereckson: /* Troubleshoot */

2023-04-25T00:16:13Z

Troubleshoot

← Older revision		Revision as of 00:16, 25 April 2023
Line 162:		Line 162:

	Also, if this error occurs often, we could automate it, discussion for that at [https://devcentral.nasqueron.org/T1816 T1816]		Also, if this error occurs often, we could automate it, discussion for that at [https://devcentral.nasqueron.org/T1816 T1816]

			=== Vault ===

			Containers based on Sentry needs access to Vault to operate. That means 3 things:

			# Private network works, ie Complector can be reached from Sentry host machine to 172.27.27.7
			# Vault is up and unsealed
			# Certificate is valid

			For example, if the Vault certificate is expired, you can have the following failure in the log:

			<syntaxhighlight lang="console">
			$ docker logs sentry_web
			!! Configuration error: ConfigurationError("SSLError: HTTPSConnectionPool(host='172.27.27.7', port=8200): Max retries exceeded with url: /v1/auth/approle/login (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')))")
			</syntaxhighlight>

Dereckson: /* Kafka */

2023-04-02T11:30:52Z

Kafka

← Older revision		Revision as of 11:30, 2 April 2023
Line 160:		Line 160:

	If you write a kafka wrapper syntax could be <code>kafka [network] <kafka server> <command></code> perhaps?		If you write a kafka wrapper syntax could be <code>kafka [network] <kafka server> <command></code> perhaps?

			Also, if this error occurs often, we could automate it, discussion for that at [https://devcentral.nasqueron.org/T1816 T1816]

Dereckson: /* Kafka */

2023-03-27T16:06:04Z

Kafka

@@ Line 88: / Line 88: @@
 A trouble shoot page exists in Sentry dev documentation, we followed successfully the out-of-sync Kafka procedure for example:
 https://develop.sentry.dev/self-hosted/troubleshooting/
 === Kafka ===

Dereckson at 15:59, 27 March 2023

2023-03-27T15:59:48Z

← Older revision		Revision as of 15:59, 27 March 2023
Line 81:		Line 81:
	\| [https://devcentral.nasqueron.org/D2881 D2881] \|\| [https://github.com/getsentry/self-hosted/commit/06662c6cdc8573f81b6a128c7bdb815efd5040ed 06662c6cdc85] \|\| 23.2.0		\| [https://devcentral.nasqueron.org/D2881 D2881] \|\| [https://github.com/getsentry/self-hosted/commit/06662c6cdc8573f81b6a128c7bdb815efd5040ed 06662c6cdc85] \|\| 23.2.0
	\|}		\|}

			== Troubleshoot ==

			=== Resources ===

			A trouble shoot page exists in Sentry dev documentation, we followed successfully the out-of-sync Kafka procedure for example:
			https://develop.sentry.dev/self-hosted/troubleshooting/

			=== Kafka ===

			After a memory issue on docker-002 and a hard reboot, the entry_post_process_forwarder_transactions and sentry_post_process_forwarder_errors containers exited on start with the following log message:

			arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}

			The out of sync procedure in [https://develop.sentry.dev/self-hosted/troubleshooting/ Sentry troubleshooting doc] works fine to solve that.

			As we don't have a kafka wrapper, you can run commands directly on a Kafka instance, or spawn a new troubleshoot container on the same network. Example of the first solution:

			<syntaxhighlight lang="console">
			$ docker exec -it sentry_kafka bash
			$ kafka-consumer-groups --bootstrap-server localhost:9092 --group snuba-post-processor -describe

			Consumer group 'snuba-post-processor' has no active members.

			GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
			snuba-post-processor events 0 43 44 1 - - -
			snuba-post-processor transactions 0 467 476 9 - - -

			$ kafka-consumer-groups --bootstrap-server localhost:9092 --group snuba-post-processor --topic events --reset-offsets --to-latest --execute

			GROUP TOPIC PARTITION NEW-OFFSET
			snuba-post-processor events 0 44

			$ kafka-consumer-groups --bootstrap-server localhost:9092 --group snuba-post-processor --topic transactions --reset-offsets --to-latest --execute

			GROUP TOPIC PARTITION NEW-OFFSET
			snuba-post-processor transactions 0 476

			$ kafka-consumer-groups --bootstrap-server localhost:9092 --group snuba-post-processor -describe

			Consumer group 'snuba-post-processor' has no active members.

			GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
			snuba-post-processor events 0 44 44 0 - - -
			snuba-post-processor transactions 0 476 476 0 - - -
			</syntaxhighlight>

			In the second case, you'll need to replace <code>--bootstrap-server localhost:9092</code> by the correct way to reach a Kafka server, for example <code>--bootstrap-server sentry_kafka:9092</code>. If it doesn't work, ensure you're on the same network.

			If you write a kafka wrapper syntax could be <code>kafka [network] <kafka server> <command></code> perhaps?

Dereckson: Update process

2023-03-25T09:50:13Z

Update process

← Older revision		Revision as of 09:50, 25 March 2023
Line 56:		Line 56:
	$ deploy-container kafka		$ deploy-container kafka
	</syntaxhighlight>		</syntaxhighlight>

			== How to upgrade Sentry? ==

			On Docker Hub or through Dwellers and our local registry, build a new nasqueron/sentry image.

			On the Docker engine, perform database migration first, then upgrade the container:
			# Pull the new image: <code>docker pull nasqueron/sentry</code>
			# Run migrations: <code>sentry nasqueron upgrade</code> (should we stop Sentry containers first?)
			# Deploy containers: <code>deploy-container sentry</code>

	== Deployment of Sentry containers ==		== Deployment of Sentry containers ==

Dereckson: /* Deployment of Sentry containers */ D2907

2023-03-25T09:20:24Z

Deployment of Sentry containers: D2907

← Older revision		Revision as of 09:20, 25 March 2023
Line 68:		Line 68:
	! rOPS !! getsentry/self-hosted !! Sentry version		! rOPS !! getsentry/self-hosted !! Sentry version
	\|-		\|-
	\| [https://devcentral.nasqueron.org/~~D2881~~ D2907] \|\| [https://github.com/getsentry/self-hosted/commit/4d318761a84cae531f3d13cbb4420e640fa660f2 4d318761a84c] \|\| 23.4.0.dev		\| [https://devcentral.nasqueron.org/D2907 D2907] \|\| [https://github.com/getsentry/self-hosted/commit/4d318761a84cae531f3d13cbb4420e640fa660f2 4d318761a84c] \|\| 23.4.0.dev
	\|-		\|-
	\| [https://devcentral.nasqueron.org/D2881 D2881] \|\| [https://github.com/getsentry/self-hosted/commit/06662c6cdc8573f81b6a128c7bdb815efd5040ed 06662c6cdc85] \|\| 23.2.0		\| [https://devcentral.nasqueron.org/D2881 D2881] \|\| [https://github.com/getsentry/self-hosted/commit/06662c6cdc8573f81b6a128c7bdb815efd5040ed 06662c6cdc85] \|\| 23.2.0
	\|}		\|}

Dereckson: Upgrade to 23.4.0.dev

2023-03-25T09:19:15Z

Upgrade to 23.4.0.dev

← Older revision		Revision as of 09:19, 25 March 2023
Line 12:		Line 12:
	If you've deployed another instance than for Nasqueron, replace it by the realm name.		If you've deployed another instance than for Nasqueron, replace it by the realm name.

	The following commands are available in 9.0:		The following commands are available in 23.3:

	cleanup Delete a portion of trailing data based on...		cleanup Delete a portion of trailing data based on creation date.
	~~cleanup_chunk~~		config Manage runtime config options.
	config Manage runtime config options.		createuser Create a new user.
	createuser Create a new user.		devserver Starts a lightweight web server for development.
	devserver Starts a lightweight web server for...		devservices Manage dependent development services required for Sentry.
	django Execute Django subcommands.		django Execute Django subcommands.
	exec Execute a script.		exec Execute a script.
	export Exports core metadata for the Sentry...		execfile Execute a script.
	files Manage files from filestore.		export Exports core metadata for the Sentry installation.
	help Show this message and exit.		files Manage files from filestore.
	import Imports data from a Sentry export.		help Show this message and exit.
	init Initialize new configuration directory.		import Imports data from a Sentry export.
	permissions Manage Permissions for Users.		init Initialize new configuration directory.
	plugins Manage Sentry plugins.		killswitches Manage killswitches for ingestion pipeline.
	queues Manage Sentry queues.		migrations Manage migrations.
	repair Attempt to repair any invalid data.		performance Performance utilities
	run Run a service.		permissions Manage Permissions for Users.
	shell Run a Python interactive interpreter.		plugins Manage Sentry plugins.
	start DEPRECATED see `sentry run` instead.		queues Manage Sentry queues.
	tsdb Tools for interacting with the time series...		repair Attempt to repair any invalid data.
	upgrade Perform any pending database migrations and...		run Run a service.
			sendmail Sends emails from the default notification mail address.
			shell Run a Python interactive interpreter.
			spans Span utilities
			start DEPRECATED see `sentry run` instead.
			tsdb Tools for interacting with the time series database.
			upgrade Perform any pending database migrations and upgrades.
			write-hashes Runs span hash grouping on event data in the supplied filename using the...

	== Kafka and ZooKeeper ==		== Kafka and ZooKeeper ==
Line 60:		Line 67:
	\|-		\|-
	! rOPS !! getsentry/self-hosted !! Sentry version		! rOPS !! getsentry/self-hosted !! Sentry version
			\|-
			\| [https://devcentral.nasqueron.org/D2881 D2907] \|\| [https://github.com/getsentry/self-hosted/commit/4d318761a84cae531f3d13cbb4420e640fa660f2 4d318761a84c] \|\| 23.4.0.dev
	\|-		\|-
	\| [https://devcentral.nasqueron.org/D2881 D2881] \|\| [https://github.com/getsentry/self-hosted/commit/06662c6cdc8573f81b6a128c7bdb815efd5040ed 06662c6cdc85] \|\| 23.2.0		\| [https://devcentral.nasqueron.org/D2881 D2881] \|\| [https://github.com/getsentry/self-hosted/commit/06662c6cdc8573f81b6a128c7bdb815efd5040ed 06662c6cdc85] \|\| 23.2.0
	\|}		\|}