{"componentChunkName":"component---src-templates-platform-tsx","path":"/platforms/python/guides/pyspark/","result":{"data":{"file":{"id":"eead4f25-0327-521c-90cc-a10a9ff92429","relativePath":"python/guides/pyspark/index.mdx","sourceInstanceName":"platforms","childMarkdownRemark":null,"childMdx":{"body":"function _extends() { _extends = Object.assign || function (target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; }; return _extends.apply(this, arguments); }\n\nfunction _objectWithoutProperties(source, excluded) { if (source == null) return {}; var target = _objectWithoutPropertiesLoose(source, excluded); var key, i; if (Object.getOwnPropertySymbols) { var sourceSymbolKeys = Object.getOwnPropertySymbols(source); for (i = 0; i < sourceSymbolKeys.length; i++) { key = sourceSymbolKeys[i]; if (excluded.indexOf(key) >= 0) continue; if (!Object.prototype.propertyIsEnumerable.call(source, key)) continue; target[key] = source[key]; } } return target; }\n\nfunction _objectWithoutPropertiesLoose(source, excluded) { if (source == null) return {}; var target = {}; var sourceKeys = Object.keys(source); var key, i; for (i = 0; i < sourceKeys.length; i++) { key = sourceKeys[i]; if (excluded.indexOf(key) >= 0) continue; target[key] = source[key]; } return target; }\n\n/* @jsx mdx */\nvar _frontmatter = {\n  \"title\": \"Apache Spark\",\n  \"redirect_from\": [\"/platforms/python/pyspark/\"]\n};\n\nvar makeShortcode = function makeShortcode(name) {\n  return function MDXDefaultShortcode(props) {\n    console.warn(\"Component \" + name + \" was not imported, exported, or provided by MDXProvider as global scope\");\n    return mdx(\"div\", props);\n  };\n};\n\nvar CodeTabs = makeShortcode(\"CodeTabs\");\nvar CodeBlock = makeShortcode(\"CodeBlock\");\nvar layoutProps = {\n  _frontmatter: _frontmatter\n};\nvar MDXLayout = \"wrapper\";\nreturn function MDXContent(_ref) {\n  var components = _ref.components,\n      props = _objectWithoutProperties(_ref, [\"components\"]);\n\n  return mdx(MDXLayout, _extends({}, layoutProps, props, {\n    components: components,\n    mdxType: \"MDXLayout\"\n  }), mdx(\"p\", null, mdx(\"em\", {\n    parentName: \"p\"\n  }, \"(New in version 0.13.0)\")), mdx(\"p\", null, \"The Spark Integration adds support for the Python API for Apache Spark, \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://spark.apache.org/\"\n  }), \"PySpark\"), \".\"), mdx(\"p\", null, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"This integration is experimental and in an alpha state\"), \". The integration API may experience breaking changes in further minor versions.\"), mdx(\"h3\", {\n    \"id\": \"driver\",\n    \"style\": {\n      \"position\": \"relative\"\n    }\n  }, mdx(\"a\", _extends({\n    parentName: \"h3\"\n  }, {\n    \"href\": \"#driver\",\n    \"aria-label\": \"driver permalink\",\n    \"className\": \"anchor before\"\n  }), mdx(\"svg\", _extends({\n    parentName: \"a\"\n  }, {\n    \"viewBox\": \"0 0 24 24\",\n    \"xmlns\": \"http://www.w3.org/2000/svg\"\n  }), mdx(\"path\", _extends({\n    parentName: \"svg\"\n  }, {\n    \"d\": \"M10.879 6.05L15 1.93A5.001 5.001 0 0 1 22.071 9l-4.121 4.121a1 1 0 0 1-1.414-1.414l4.12-4.121a3 3 0 1 0-4.242-4.243l-4.121 4.121a1 1 0 1 1-1.414-1.414zm2.242 11.9L9 22.07A5 5 0 1 1 1.929 15l4.121-4.121a1 1 0 0 1 1.414 1.414l-4.12 4.121a3 3 0 1 0 4.242 4.243l4.121-4.121a1 1 0 1 1 1.414 1.414zm-8.364-.122l13.071-13.07a1 1 0 0 1 1.415 1.414L6.172 19.242a1 1 0 1 1-1.415-1.414z\",\n    \"fill\": \"currentColor\"\n  })))), \"Driver\"), mdx(\"p\", null, \"The spark driver integration is supported for Spark 2 and above.\"), mdx(\"p\", null, \"To configure the SDK, initialize it with the integration before you create a \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"SparkContext\"), \" or \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"SparkSession\"), \".\"), mdx(\"div\", {\n    \"className\": \"code-tabs-wrapper\"\n  }, mdx(CodeTabs, {\n    mdxType: \"CodeTabs\"\n  }, mdx(CodeBlock, {\n    language: \"python\",\n    title: \"\",\n    filename: \"\",\n    mdxType: \"CodeBlock\"\n  }, mdx(\"div\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"python\"\n  }), mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-python\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-python\"\n  }), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"import\"), \" sentry_sdk\\n\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"from\"), \" sentry_sdk\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"integrations\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"spark \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"import\"), \" SparkIntegration\\n\\n\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"if\"), \" __name__ \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), \"==\"), \" \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), \"\\\"__main__\\\"\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \":\"), \"\\n    sentry_sdk\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"init\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"(\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), \"\\\"___PUBLIC_DSN___\\\"\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \",\"), \" integrations\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), \"=\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"[\"), \"SparkIntegration\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"(\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \")\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"]\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \")\"), \"\\n\\n    spark \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), \"=\"), \" SparkSession\\\\\\n        \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"builder\\\\\\n        \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"appName\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"(\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), \"\\\"ExampleApp\\\"\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \")\"), \"\\\\\\n        \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"getOrCreate\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"(\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \")\"), \"\\n    \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"))))))), mdx(\"h3\", {\n    \"id\": \"worker\",\n    \"style\": {\n      \"position\": \"relative\"\n    }\n  }, mdx(\"a\", _extends({\n    parentName: \"h3\"\n  }, {\n    \"href\": \"#worker\",\n    \"aria-label\": \"worker permalink\",\n    \"className\": \"anchor before\"\n  }), mdx(\"svg\", _extends({\n    parentName: \"a\"\n  }, {\n    \"viewBox\": \"0 0 24 24\",\n    \"xmlns\": \"http://www.w3.org/2000/svg\"\n  }), mdx(\"path\", _extends({\n    parentName: \"svg\"\n  }, {\n    \"d\": \"M10.879 6.05L15 1.93A5.001 5.001 0 0 1 22.071 9l-4.121 4.121a1 1 0 0 1-1.414-1.414l4.12-4.121a3 3 0 1 0-4.242-4.243l-4.121 4.121a1 1 0 1 1-1.414-1.414zm2.242 11.9L9 22.07A5 5 0 1 1 1.929 15l4.121-4.121a1 1 0 0 1 1.414 1.414l-4.12 4.121a3 3 0 1 0 4.242 4.243l4.121-4.121a1 1 0 1 1 1.414 1.414zm-8.364-.122l13.071-13.07a1 1 0 0 1 1.415 1.414L6.172 19.242a1 1 0 1 1-1.415-1.414z\",\n    \"fill\": \"currentColor\"\n  })))), \"Worker\"), mdx(\"p\", null, \"The spark worker integration is supported for Spark 2.4.0 and above.\"), mdx(\"p\", null, \"Create a file called \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"sentry-daemon.py\"), \" with the following content:\"), mdx(\"div\", {\n    \"className\": \"code-tabs-wrapper\"\n  }, mdx(CodeTabs, {\n    mdxType: \"CodeTabs\"\n  }, mdx(CodeBlock, {\n    language: \"python\",\n    title: \"\",\n    filename: \"sentry-daemon.py\",\n    mdxType: \"CodeBlock\"\n  }, mdx(\"div\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"python\"\n  }), mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-python\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-python\"\n  }), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"import\"), \" sentry_sdk\\n\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"from\"), \" sentry_sdk\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"integrations\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"spark \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"import\"), \" SparkWorkerIntegration\\n\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"import\"), \" pyspark\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"daemon \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"as\"), \" original_daemon\\n\\n\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), \"if\"), \" __name__ \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), \"==\"), \" \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), \"'__main__'\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \":\"), \"\\n    sentry_sdk\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"init\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"(\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), \"\\\"___PUBLIC_DSN___\\\"\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \",\"), \" integrations\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), \"=\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"[\"), \"SparkWorkerIntegration\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"(\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \")\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"]\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \")\"), \"\\n    original_daemon\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \".\"), \"manager\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"(\"), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \")\"))))))), mdx(\"p\", null, \"In your \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"spark_submit\"), \" command, add the following configuration options so the spark clusters can use the sentry integration.\"), mdx(\"table\", null, mdx(\"thead\", {\n    parentName: \"table\"\n  }, mdx(\"tr\", {\n    parentName: \"thead\"\n  }, mdx(\"th\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"Command Line Options\"), mdx(\"th\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"Parameter\"), mdx(\"th\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"Usage\"))), mdx(\"tbody\", {\n    parentName: \"table\"\n  }, mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"--py-files\"), mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"sentry_daemon.py\"), mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"Sends the \", mdx(\"inlineCode\", {\n    parentName: \"td\"\n  }, \"sentry_daemon.py\"), \" file to your Spark clusters\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"--conf\"), mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"spark.python.use.daemon=true\"), mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"Configures Spark to use a daemon to execute it's Python workers\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"--conf\"), mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"spark.python.daemon.module=sentry_daemon\"), mdx(\"td\", _extends({\n    parentName: \"tr\"\n  }, {\n    \"align\": null\n  }), \"Configures Spark to use the sentry custom daemon\")))), mdx(\"div\", {\n    \"className\": \"code-tabs-wrapper\"\n  }, mdx(CodeTabs, {\n    mdxType: \"CodeTabs\"\n  }, mdx(CodeBlock, {\n    language: \"bash\",\n    title: \"\",\n    filename: \"\",\n    mdxType: \"CodeBlock\"\n  }, mdx(\"div\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"bash\"\n  }), mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-bash\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-bash\"\n  }), \"./bin/spark-submit \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"\\\\\"), \"\\n    --py-files sentry_daemon.py \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"\\\\\"), \"\\n    --conf spark.python.use.daemon\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), \"=\"), \"true \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"\\\\\"), \"\\n    --conf spark.python.daemon.module\", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), \"=\"), \"sentry_daemon \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), \"\\\\\"), \"\\n    example-spark-job.py\")))))), mdx(\"h2\", {\n    \"id\": \"behavior\",\n    \"style\": {\n      \"position\": \"relative\"\n    }\n  }, mdx(\"a\", _extends({\n    parentName: \"h2\"\n  }, {\n    \"href\": \"#behavior\",\n    \"aria-label\": \"behavior permalink\",\n    \"className\": \"anchor before\"\n  }), mdx(\"svg\", _extends({\n    parentName: \"a\"\n  }, {\n    \"viewBox\": \"0 0 24 24\",\n    \"xmlns\": \"http://www.w3.org/2000/svg\"\n  }), mdx(\"path\", _extends({\n    parentName: \"svg\"\n  }, {\n    \"d\": \"M10.879 6.05L15 1.93A5.001 5.001 0 0 1 22.071 9l-4.121 4.121a1 1 0 0 1-1.414-1.414l4.12-4.121a3 3 0 1 0-4.242-4.243l-4.121 4.121a1 1 0 1 1-1.414-1.414zm2.242 11.9L9 22.07A5 5 0 1 1 1.929 15l4.121-4.121a1 1 0 0 1 1.414 1.414l-4.12 4.121a3 3 0 1 0 4.242 4.243l4.121-4.121a1 1 0 1 1 1.414 1.414zm-8.364-.122l13.071-13.07a1 1 0 0 1 1.415 1.414L6.172 19.242a1 1 0 1 1-1.415-1.414z\",\n    \"fill\": \"currentColor\"\n  })))), \"Behavior\"), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"You must have the sentry python sdk installed on all your clusters to use the Spark integration. The easiest way to do this is to run an initialization script on all your clusters:\")), mdx(\"div\", {\n    \"className\": \"code-tabs-wrapper\"\n  }, mdx(CodeTabs, {\n    mdxType: \"CodeTabs\"\n  }, mdx(CodeBlock, {\n    language: \"bash\",\n    title: \"\",\n    filename: \"\",\n    mdxType: \"CodeBlock\"\n  }, mdx(\"div\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"bash\"\n  }), mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-bash\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-bash\"\n  }), \"easy_install pip\\npip \", mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), \"install\"), \" --upgrade sentry-sdk\")))))), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, \"In order to access certain tags (\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"app_name\"), \", \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"application_id\"), \"), the worker integration requires the driver integration to also be active.\")), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, \"The worker integration only works on UNIX-based systems due to the daemon process using signals for child management.\"))), mdx(\"h2\", {\n    \"id\": \"google-cloud-dataproc\",\n    \"style\": {\n      \"position\": \"relative\"\n    }\n  }, mdx(\"a\", _extends({\n    parentName: \"h2\"\n  }, {\n    \"href\": \"#google-cloud-dataproc\",\n    \"aria-label\": \"google cloud dataproc permalink\",\n    \"className\": \"anchor before\"\n  }), mdx(\"svg\", _extends({\n    parentName: \"a\"\n  }, {\n    \"viewBox\": \"0 0 24 24\",\n    \"xmlns\": \"http://www.w3.org/2000/svg\"\n  }), mdx(\"path\", _extends({\n    parentName: \"svg\"\n  }, {\n    \"d\": \"M10.879 6.05L15 1.93A5.001 5.001 0 0 1 22.071 9l-4.121 4.121a1 1 0 0 1-1.414-1.414l4.12-4.121a3 3 0 1 0-4.242-4.243l-4.121 4.121a1 1 0 1 1-1.414-1.414zm2.242 11.9L9 22.07A5 5 0 1 1 1.929 15l4.121-4.121a1 1 0 0 1 1.414 1.414l-4.12 4.121a3 3 0 1 0 4.242 4.243l4.121-4.121a1 1 0 1 1 1.414 1.414zm-8.364-.122l13.071-13.07a1 1 0 0 1 1.415 1.414L6.172 19.242a1 1 0 1 1-1.415-1.414z\",\n    \"fill\": \"currentColor\"\n  })))), \"Google Cloud Dataproc\"), mdx(\"p\", null, \"This integration can be set up to be used with \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://cloud.google.com/dataproc/\"\n  }), \"Google Cloud Dataproc\"), \". It is recommended that Cloud Dataproc image version 1.4 be used as it comes with Spark 2.4 (required by the worker integration).\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, \"Set up an \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/init-actions\"\n  }), \"Initialization action\"), \" to install the \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"sentry-sdk\"), \" on your Dataproc cluster.\")), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, \"Add the driver integration to your main python file submitted in in the job submit screen\")), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, \"Add the \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"sentry_daemon.py\"), \" under \", mdx(\"i\", null, \"Additional python files\"), \" in the job submit screen. You must first upload the daemon file to a bucket to access it.\")), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, \"Add the configuration properties listed above, \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"spark.python.use.daemon=true\"), \" and \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"spark.python.daemon.module=sentry_daemon\"), \" in the job submit screen.\"))));\n}\n;\nMDXContent.isMDXComponent = true;","tableOfContents":{"items":[{"items":[{"url":"#driver","title":"Driver"},{"url":"#worker","title":"Worker"}]},{"url":"#behavior","title":"Behavior"},{"url":"#google-cloud-dataproc","title":"Google Cloud Dataproc"}]},"internal":{"type":"Mdx"}}}},"pageContext":{"excerpt":"(New in version 0.13.0) The Spark Integration adds support for the Python API for Apache Spark,  PySpark . This integration is experimental and in an alpha state . The integration API may experience breaking changes in further minor versions. Driver The spark driver integration is supported for Spark 2 and above. To configure the SDK, initialize it with the integration before you create a  SparkContext  or  SparkSession . Worker The spark worker integration is supported for Spark 2.4.0 and above. Create a file called  sentry-daemon.py  with the following content: In your  spark_submit  command, add the following configuration options so the spark clusters can use the sentry integration. Command Line Options Parameter Usage --py-files sentry_daemon.py Sends the  sentry_daemon.py  file to your Spark clusters --conf spark.python.use.daemon=true Configures Spark to use a daemon to execute it's Python workers --conf spark.python.daemon.module=sentry_daemon Configures Spark to use the sentry custom daemon Behavior You must have the sentry python sdk installed on all your clusters to use the Spark integration. The easiest way to do this is to run an initialization script on all your clusters: In order to access certain tags ( app_name ,  application_id ), the worker integration requires the driver integration to also be active. The worker integration only works on UNIX-based systems due to the daemon process using signals for child management. Google Cloud Dataproc This integration can be set up to be used with  Google Cloud Dataproc . It is recommended that Cloud Dataproc image version 1.4 be used as it comes with Spark 2.4 (required by the worker integration). Set up an  Initialization action  to install the  sentry-sdk  on your Dataproc cluster. Add the driver integration to your main python file submitted in in the job submit screen Add the  sentry_daemon.py  under  Additional python files  in the job submit screen. You must first upload the daemon file to a bucket to access it. Add the configuration properties listed above,  spark.python.use.daemon=true  and  spark.python.daemon.module=sentry_daemon  in the job submit screen.","title":"Apache Spark","redirect_from":["/platforms/python/pyspark/"],"platform":{"name":"python","title":"Python"},"guide":{"name":"pyspark","title":"Apache Spark"},"id":"eead4f25-0327-521c-90cc-a10a9ff92429"}},"staticQueryHashes":["1218203755","1222113826","1222113826","1766336459","2158593473","2404336828","2472290386","2764967025","3818502851","4015007367","4192517163","4264099332","518019976"]}