{
  "paragraphs": [
    {
      "title": "Introduction",
      "text": "%md\n\nThis is a tutorial of how to use Spark MLlib in Zeppelin, we have 2 examples in this note:\n\n* Linear regression, we generate some random data and use a linear regression to fit this data. We use bokeh here to visualize the data and the fitted model.  Besides training, we also visualize the loss value over iteration.\n* Logstic regression, we use the official `sample_binary_classification_data` of spark as the training data. Besides training, we also visualize the loss value over iteration.\n",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:31:55.260",
      "progress": 0,
      "config": {
        "runOnSelectionChange": true,
        "title": true,
        "checkEmpty": true,
        "colWidth": 12.0,
        "fontSize": 9.0,
        "enabled": true,
        "results": {},
        "editorSetting": {
          "language": "text",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "editorMode": "ace/mode/text",
        "editorHide": true
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "HTML",
            "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis is a tutorial of how to use Spark MLlib in Zeppelin, we have 2 examples in this note:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eLinear regression, we generate some random data and use a linear regression to fit this data. We use bokeh here to visualize the data and the fitted model.  Besides training, we also visualize the loss value over iteration.\u003c/li\u003e\n\u003cli\u003eLogstic regression, we use the official \u003ccode\u003esample_binary_classification_data\u003c/code\u003e of spark as the training data. Besides training, we also visualize the loss value over iteration.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003c/div\u003e"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {},
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1579424966763_-908073771",
      "id": "paragraph_1579424966763_-908073771",
      "dateCreated": "2020-01-19 17:09:26.763",
      "dateStarted": "2021-07-26 04:31:55.265",
      "dateFinished": "2021-07-26 04:31:57.477",
      "status": "FINISHED"
    },
    {
      "title": "Initialize bokeh for visualization",
      "text": "%spark.pyspark\n\n\nimport numpy as np\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\n\noutput_notebook()\n\n",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:31:57.565",
      "progress": 0,
      "config": {
        "colWidth": 12.0,
        "editorMode": "ace/mode/python",
        "results": {},
        "enabled": true,
        "editorSetting": {
          "language": "python",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "fontSize": 9.0,
        "runOnSelectionChange": true,
        "title": true,
        "checkEmpty": true
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "HTML",
            "data": "\n    \u003cdiv class\u003d\"bk-root\"\u003e\n        \u003ca href\u003d\"https://bokeh.org\" target\u003d\"_blank\" class\u003d\"bk-logo bk-logo-small bk-logo-notebook\"\u003e\u003c/a\u003e\n        \u003cspan id\u003d\"1002\"\u003eLoading BokehJS ...\u003c/span\u003e\n    \u003c/div\u003e\n"
          },
          {
            "type": "HTML",
            "data": "\u003cscript\u003e \n(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  var force \u003d true;\n\n  if (typeof root._bokeh_onload_callbacks \u003d\u003d\u003d \"undefined\" || force \u003d\u003d\u003d true) {\n    root._bokeh_onload_callbacks \u003d [];\n    root._bokeh_is_loading \u003d undefined;\n  }\n\n  var JS_MIME_TYPE \u003d \u0027application/javascript\u0027;\n  var HTML_MIME_TYPE \u003d \u0027text/html\u0027;\n  var EXEC_MIME_TYPE \u003d \u0027application/vnd.bokehjs_exec.v0+json\u0027;\n  var CLASS_NAME \u003d \u0027output_bokeh rendered_html\u0027;\n\n  /**\n   * Render data to the DOM node\n   */\n  function render(props, node) {\n    var script \u003d document.createElement(\"script\");\n    node.appendChild(script);\n  }\n\n  /**\n   * Handle when an output is cleared or removed\n   */\n  function handleClearOutput(event, handle) {\n    var cell \u003d handle.cell;\n\n    var id \u003d cell.output_area._bokeh_element_id;\n    var server_id \u003d cell.output_area._bokeh_server_id;\n    // Clean up Bokeh references\n    if (id !\u003d null \u0026\u0026 id in Bokeh.index) {\n      Bokeh.index[id].model.document.clear();\n      delete Bokeh.index[id];\n    }\n\n    if (server_id !\u003d\u003d undefined) {\n      // Clean up Bokeh references\n      var cmd \u003d \"from bokeh.io.state import curstate; print(curstate().uuid_to_server[\u0027\" + server_id + \"\u0027].get_sessions()[0].document.roots[0]._id)\";\n      cell.notebook.kernel.execute(cmd, {\n        iopub: {\n          output: function(msg) {\n            var id \u003d msg.content.text.trim();\n            if (id in Bokeh.index) {\n              Bokeh.index[id].model.document.clear();\n              delete Bokeh.index[id];\n            }\n          }\n        }\n      });\n      // Destroy server and session\n      var cmd \u003d \"import bokeh.io.notebook as ion; ion.destroy_server(\u0027\" + server_id + \"\u0027)\";\n      cell.notebook.kernel.execute(cmd);\n    }\n  }\n\n  /**\n   * Handle when a new output is added\n   */\n  function handleAddOutput(event, handle) {\n    var output_area \u003d handle.output_area;\n    var output \u003d handle.output;\n\n    // limit handleAddOutput to display_data with EXEC_MIME_TYPE content only\n    if ((output.output_type !\u003d \"display_data\") || (!Object.prototype.hasOwnProperty.call(output.data, EXEC_MIME_TYPE))) {\n      return\n    }\n\n    var toinsert \u003d output_area.element.find(\".\" + CLASS_NAME.split(\u0027 \u0027)[0]);\n\n    if (output.metadata[EXEC_MIME_TYPE][\"id\"] !\u003d\u003d undefined) {\n      toinsert[toinsert.length - 1].firstChild.textContent \u003d output.data[JS_MIME_TYPE];\n      // store reference to embed id on output_area\n      output_area._bokeh_element_id \u003d output.metadata[EXEC_MIME_TYPE][\"id\"];\n    }\n    if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !\u003d\u003d undefined) {\n      var bk_div \u003d document.createElement(\"div\");\n      bk_div.innerHTML \u003d output.data[HTML_MIME_TYPE];\n      var script_attrs \u003d bk_div.children[0].attributes;\n      for (var i \u003d 0; i \u003c script_attrs.length; i++) {\n        toinsert[toinsert.length - 1].firstChild.setAttribute(script_attrs[i].name, script_attrs[i].value);\n        toinsert[toinsert.length - 1].firstChild.textContent \u003d bk_div.children[0].textContent\n      }\n      // store reference to server id on output_area\n      output_area._bokeh_server_id \u003d output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n    }\n  }\n\n  function register_renderer(events, OutputArea) {\n\n    function append_mime(data, metadata, element) {\n      // create a DOM node to render to\n      var toinsert \u003d this.create_output_subarea(\n        metadata,\n        CLASS_NAME,\n        EXEC_MIME_TYPE\n      );\n      this.keyboard_manager.register_events(toinsert);\n      // Render to node\n      var props \u003d {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n      render(props, toinsert[toinsert.length - 1]);\n      element.append(toinsert);\n      return toinsert\n    }\n\n    /* Handle when an output is cleared or removed */\n    events.on(\u0027clear_output.CodeCell\u0027, handleClearOutput);\n    events.on(\u0027delete.Cell\u0027, handleClearOutput);\n\n    /* Handle when a new output is added */\n    events.on(\u0027output_added.OutputArea\u0027, handleAddOutput);\n\n    /**\n     * Register the mime type and append_mime function with output_area\n     */\n    OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n      /* Is output safe? */\n      safe: true,\n      /* Index of renderer in `output_area.display_order` */\n      index: 0\n    });\n  }\n\n  // register the mime type if in Jupyter Notebook environment and previously unregistered\n  if (root.Jupyter !\u003d\u003d undefined) {\n    var events \u003d require(\u0027base/js/events\u0027);\n    var OutputArea \u003d require(\u0027notebook/js/outputarea\u0027).OutputArea;\n\n    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) \u003d\u003d -1) {\n      register_renderer(events, OutputArea);\n    }\n  }\n\n  \n  if (typeof (root._bokeh_timeout) \u003d\u003d\u003d \"undefined\" || force \u003d\u003d\u003d true) {\n    root._bokeh_timeout \u003d Date.now() + 5000;\n    root._bokeh_failed_load \u003d false;\n  }\n\n  var NB_LOAD_WARNING \u003d {\u0027data\u0027: {\u0027text/html\u0027:\n     \"\u003cdiv style\u003d\u0027background-color: #fdd\u0027\u003e\\n\"+\n     \"\u003cp\u003e\\n\"+\n     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n     \"\u003c/p\u003e\\n\"+\n     \"\u003cul\u003e\\n\"+\n     \"\u003cli\u003ere-rerun `output_notebook()` to attempt to load from CDN again, or\u003c/li\u003e\\n\"+\n     \"\u003cli\u003euse INLINE resources instead, as so:\u003c/li\u003e\\n\"+\n     \"\u003c/ul\u003e\\n\"+\n     \"\u003ccode\u003e\\n\"+\n     \"from bokeh.resources import INLINE\\n\"+\n     \"output_notebook(resources\u003dINLINE)\\n\"+\n     \"\u003c/code\u003e\\n\"+\n     \"\u003c/div\u003e\"}};\n\n  function display_loaded() {\n    var el \u003d document.getElementById(\"1002\");\n    if (el !\u003d null) {\n      el.textContent \u003d \"BokehJS is loading...\";\n    }\n    if (root.Bokeh !\u003d\u003d undefined) {\n      if (el !\u003d null) {\n        el.textContent \u003d \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n      }\n    } else if (Date.now() \u003c root._bokeh_timeout) {\n      setTimeout(display_loaded, 100)\n    }\n  }\n\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback !\u003d null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, callback) {\n    if (css_urls \u003d\u003d null) css_urls \u003d [];\n    if (js_urls \u003d\u003d null) js_urls \u003d [];\n\n    root._bokeh_onload_callbacks.push(callback);\n    if (root._bokeh_is_loading \u003e 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls \u003d\u003d null || js_urls.length \u003d\u003d\u003d 0) {\n      run_callbacks();\n      return null;\n    }\n    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    root._bokeh_is_loading \u003d css_urls.length + js_urls.length;\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading \u003d\u003d\u003d 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n\n    function on_error(url) {\n      console.error(\"failed to load \" + url);\n    }\n\n    for (let i \u003d 0; i \u003c css_urls.length; i++) {\n      const url \u003d css_urls[i];\n      const element \u003d document.createElement(\"link\");\n      element.onload \u003d on_load;\n      element.onerror \u003d on_error.bind(null, url);\n      element.rel \u003d \"stylesheet\";\n      element.type \u003d \"text/css\";\n      element.href \u003d url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }\n\n    const hashes \u003d {\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\": \"dM3QQsP+wXdHg42wTqW85BjZQdLNNIXqlPw/BgKoExPmTG7ZLML4EGqLMfqHT6ON\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\": \"8x57I4YuIfu8XyZfFo0XVr2WAT8EK4rh/uDe3wF7YuW2FNUSNEpJbsPaB1nJ2fz2\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\": \"3QTqdz9LyAm2i0sG5XTePsHec3UHWwVsrOL68SYRoAXsafvfAyqtQ+h440+qIBhS\"};\n\n    for (let i \u003d 0; i \u003c js_urls.length; i++) {\n      const url \u003d js_urls[i];\n      const element \u003d document.createElement(\u0027script\u0027);\n      element.onload \u003d on_load;\n      element.onerror \u003d on_error.bind(null, url);\n      element.async \u003d false;\n      element.src \u003d url;\n      if (url in hashes) {\n        element.crossOrigin \u003d \"anonymous\";\n        element.integrity \u003d \"sha384-\" + hashes[url];\n      }\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element \u003d document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  \n  var js_urls \u003d [\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\"];\n  var css_urls \u003d [];\n  \n\n  var inline_js \u003d [\n    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\n    function(Bokeh) {\n    \n    \n    }\n  ];\n\n  function run_inline_js() {\n    \n    if (root.Bokeh !\u003d\u003d undefined || force \u003d\u003d\u003d true) {\n      \n    for (var i \u003d 0; i \u003c inline_js.length; i++) {\n      inline_js[i].call(root, root.Bokeh);\n    }\n    if (force \u003d\u003d\u003d true) {\n        display_loaded();\n      }} else if (Date.now() \u003c root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load \u003d true;\n    } else if (force !\u003d\u003d true) {\n      var cell \u003d $(document.getElementById(\"1002\")).parents(\u0027.cell\u0027).data().cell;\n      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n    }\n\n  }\n\n  if (root._bokeh_is_loading \u003d\u003d\u003d 0) {\n    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n    run_inline_js();\n  } else {\n    load_libs(css_urls, js_urls, function() {\n      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n      run_inline_js();\n    });\n  }\n}(window)); \u003c/script\u003e\n"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {},
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1578387637676_907388241",
      "id": "20170621-112503_896367416",
      "dateCreated": "2020-01-07 17:00:37.676",
      "dateStarted": "2021-07-26 04:31:57.569",
      "dateFinished": "2021-07-26 04:32:15.826",
      "status": "FINISHED"
    },
    {
      "title": "Generate Data",
      "text": "%spark.pyspark\n\nimport numpy as np\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\n\n\nnum \u003d 1000\nx \u003d np.linspace(0, 10, num)\ny \u003d 2 * x + np.random.normal(0,4, num)\n\np \u003d figure()\np.circle(x, y)\nshow(p)",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:32:15.892",
      "progress": 0,
      "config": {
        "editorSetting": {
          "language": "python",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "colWidth": 6.0,
        "editorMode": "ace/mode/python",
        "title": true,
        "results": {},
        "enabled": true,
        "fontSize": 9.0,
        "runOnSelectionChange": true,
        "checkEmpty": true
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "HTML",
            "data": "\n\n\n\n\n\n  \u003cdiv class\u003d\"bk-root\" id\u003d\"f10d9be9-2e8a-43ce-9d36-1220b5a7a5e4\" data-root-id\u003d\"1003\"\u003e\u003c/div\u003e\n\n"
          },
          {
            "type": "HTML",
            "data": "\u003cscript\u003e (function(root) {\n  function embed_document(root) {\n    \n  var docs_json \u003d {\"e803d4a1-2663-44a2-bbb5-7490333e25ae\":{\"defs\":[],\"roots\":{\"references\":[{\"attributes\":{\"below\":[{\"id\":\"1012\"}],\"center\":[{\"id\":\"1015\"},{\"id\":\"1019\"}],\"left\":[{\"id\":\"1016\"}],\"renderers\":[{\"id\":\"1037\"}],\"title\":{\"id\":\"1040\"},\"toolbar\":{\"id\":\"1027\"},\"x_range\":{\"id\":\"1004\"},\"x_scale\":{\"id\":\"1008\"},\"y_range\":{\"id\":\"1006\"},\"y_scale\":{\"id\":\"1010\"}},\"id\":\"1003\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{},\"id\":\"1006\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1047\",\"type\":\"UnionRenderers\"},{\"attributes\":{},\"id\":\"1045\",\"type\":\"AllLabels\"},{\"attributes\":{},\"id\":\"1004\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1021\",\"type\":\"WheelZoomTool\"},{\"attributes\":{},\"id\":\"1025\",\"type\":\"HelpTool\"},{\"attributes\":{},\"id\":\"1040\",\"type\":\"Title\"},{\"attributes\":{},\"id\":\"1008\",\"type\":\"LinearScale\"},{\"attributes\":{\"axis\":{\"id\":\"1016\"},\"dimension\":1,\"ticker\":null},\"id\":\"1019\",\"type\":\"Grid\"},{\"attributes\":{\"overlay\":{\"id\":\"1026\"}},\"id\":\"1022\",\"type\":\"BoxZoomTool\"},{\"attributes\":{},\"id\":\"1048\",\"type\":\"Selection\"},{\"attributes\":{\"formatter\":{\"id\":\"1043\"},\"major_label_policy\":{\"id\":\"1042\"},\"ticker\":{\"id\":\"1017\"}},\"id\":\"1016\",\"type\":\"LinearAxis\"},{\"attributes\":{\"formatter\":{\"id\":\"1046\"},\"major_label_policy\":{\"id\":\"1045\"},\"ticker\":{\"id\":\"1013\"}},\"id\":\"1012\",\"type\":\"LinearAxis\"},{\"attributes\":{\"fill_color\":{\"value\":\"#1f77b4\"},\"line_color\":{\"value\":\"#1f77b4\"},\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1035\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"1010\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1020\",\"type\":\"PanTool\"},{\"attributes\":{},\"id\":\"1042\",\"type\":\"AllLabels\"},{\"attributes\":{\"data\":{\"x\":{\"__ndarray__\":\"[encoded data, 10667 characters]\u003d\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[1000]},\"y\":{\"__ndarray__\":\"[encoded data, 10667 characters]\u003d\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[1000]}},\"selected\":{\"id\":\"1048\"},\"selection_policy\":{\"id\":\"1047\"}},\"id\":\"1034\",\"type\":\"ColumnDataSource\"},{\"attributes\":{},\"id\":\"1023\",\"type\":\"SaveTool\"},{\"attributes\":{},\"id\":\"1017\",\"type\":\"BasicTicker\"},{\"attributes\":{\"bottom_units\":\"screen\",\"fill_alpha\":0.5,\"fill_color\":\"lightgrey\",\"left_units\":\"screen\",\"level\":\"overlay\",\"line_alpha\":1.0,\"line_color\":\"black\",\"line_dash\":[4,4],\"line_width\":2,\"right_units\":\"screen\",\"syncable\":false,\"top_units\":\"screen\"},\"id\":\"1026\",\"type\":\"BoxAnnotation\"},{\"attributes\":{\"data_source\":{\"id\":\"1034\"},\"glyph\":{\"id\":\"1035\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1036\"},\"view\":{\"id\":\"1038\"}},\"id\":\"1037\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1043\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"axis\":{\"id\":\"1012\"},\"ticker\":null},\"id\":\"1015\",\"type\":\"Grid\"},{\"attributes\":{\"source\":{\"id\":\"1034\"}},\"id\":\"1038\",\"type\":\"CDSView\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1036\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"1024\",\"type\":\"ResetTool\"},{\"attributes\":{\"active_multi\":null,\"tools\":[{\"id\":\"1020\"},{\"id\":\"1021\"},{\"id\":\"1022\"},{\"id\":\"1023\"},{\"id\":\"1024\"},{\"id\":\"1025\"}]},\"id\":\"1027\",\"type\":\"Toolbar\"},{\"attributes\":{},\"id\":\"1046\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1013\",\"type\":\"BasicTicker\"}],\"root_ids\":[\"1003\"]},\"title\":\"Bokeh Application\",\"version\":\"2.3.3\"}};\n  var render_items \u003d [{\"docid\":\"e803d4a1-2663-44a2-bbb5-7490333e25ae\",\"root_ids\":[\"1003\"],\"roots\":{\"1003\":\"f10d9be9-2e8a-43ce-9d36-1220b5a7a5e4\"}}];\n  root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n\n  }\n  if (root.Bokeh !\u003d\u003d undefined) {\n    embed_document(root);\n  } else {\n    var attempts \u003d 0;\n    var timer \u003d setInterval(function(root) {\n      if (root.Bokeh !\u003d\u003d undefined) {\n        clearInterval(timer);\n        embed_document(root);\n      } else {\n        attempts++;\n        if (attempts \u003e 100) {\n          clearInterval(timer);\n          console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n        }\n      }\n    }, 10, root)\n  }\n})(window); \u003c/script\u003e\n"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {},
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1578387637677_-677869302",
      "id": "20170621-112549_1048642377",
      "dateCreated": "2020-01-07 17:00:37.677",
      "dateStarted": "2021-07-26 04:32:15.895",
      "dateFinished": "2021-07-26 04:32:16.138",
      "status": "FINISHED"
    },
    {
      "title": "Linear Regression",
      "text": "%spark.pyspark\n\n\nfrom pyspark.ml.regression import LinearRegression\nimport pandas as pd\nfrom pyspark.ml.linalg import DenseVector, Vectors, VectorUDT\nfrom pyspark.sql.functions import udf\nfrom pyspark.sql.types import UserDefinedType, StringType\n\nto_vector \u003d udf(lambda x: Vectors.dense(x), VectorUDT())\n\ndf \u003d pd.DataFrame({\u0027features\u0027: x, \u0027label\u0027: y})\ntraining \u003d spark.createDataFrame(df).withColumn(\u0027features\u0027, to_vector(\u0027features\u0027))\n\nlr \u003d LinearRegression(maxIter\u003d50, regParam\u003d0.3, elasticNetParam\u003d0.8, solver\u003d\u0027l-bfgs\u0027)\n\n# Fit the model\nlrModel \u003d lr.fit(training)\n\n# Print the coefficients and intercept for linear regression\nprint(\"Coefficients: %s\" % str(lrModel.coefficients))\nprint(\"Intercept: %s\" % str(lrModel.intercept))\n\n# Summarize the model over the training set and print out some metrics\ntrainingSummary \u003d lrModel.summary\nprint(\"numIterations: %d\" % trainingSummary.totalIterations)\nprint(\"objectiveHistory: %s\" % str(trainingSummary.objectiveHistory))\ntrainingSummary.residuals.show()\nprint(\"RMSE: %f\" % trainingSummary.rootMeanSquaredError)\nprint(\"r2: %f\" % trainingSummary.r2)\n\n\n",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:32:16.195",
      "progress": 100,
      "config": {
        "editorSetting": {
          "language": "python",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "colWidth": 6.0,
        "editorMode": "ace/mode/python",
        "title": true,
        "results": {},
        "enabled": true,
        "fontSize": 9.0,
        "runOnSelectionChange": true,
        "checkEmpty": true
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "TEXT",
            "data": "Coefficients: [1.9367837330953086]\nIntercept: 0.31946056039263304\nnumIterations: 2\nobjectiveHistory: [0.49950000000000033, 0.424180453174143, 0.18300563355387592]\n+--------------------+\n|           residuals|\n+--------------------+\n|  -2.833166305176022|\n| -0.7068594675252808|\n|  0.1322891865273667|\n|   5.456616501979674|\n|  0.8225765174346039|\n|   3.774681833437321|\n|-0.03607428902607...|\n|  -2.032649908137373|\n|  -4.083954805729507|\n|  -4.591691412436773|\n|  2.6705538935268467|\n|   2.269737051649101|\n|    2.22074748343437|\n|  3.6738694179491995|\n|  -5.647995770675787|\n| 0.44724887877671227|\n|     2.7171755410366|\n|  -2.894179997049915|\n|  2.0272052033953236|\n| -0.6040963708941917|\n+--------------------+\nonly showing top 20 rows\n\nRMSE: 3.912953\nr2: 0.692876\n"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {
        "jobUrl": {
          "propertyName": "jobUrl",
          "label": "SPARK JOB",
          "tooltip": "View in Spark web UI",
          "group": "spark",
          "values": [
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d0"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d1"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d2"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d3"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d4"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d5"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d6"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d7"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d8"
            }
          ],
          "interpreterSettingId": "spark"
        }
      },
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1578387637679_-1199864234",
      "id": "20170621-120434_191947265",
      "dateCreated": "2020-01-07 17:00:37.679",
      "dateStarted": "2021-07-26 04:32:16.198",
      "dateFinished": "2021-07-26 04:32:24.250",
      "status": "FINISHED"
    },
    {
      "title": "Linear Regression",
      "text": "%spark.pyspark\n\n\np \u003d figure()\np.circle(x, y)\n\ny_predict \u003d x * lrModel.coefficients + (lrModel.intercept)\np.line(x, y_predict, color\u003d\u0027red\u0027, line_width\u003d3)\n\nshow(p)",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:32:24.305",
      "progress": 0,
      "config": {
        "editorSetting": {
          "language": "python",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "colWidth": 6.0,
        "editorMode": "ace/mode/python",
        "title": true,
        "results": {},
        "enabled": true,
        "fontSize": 9.0,
        "runOnSelectionChange": true,
        "checkEmpty": true
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "HTML",
            "data": "\n\n\n\n\n\n  \u003cdiv class\u003d\"bk-root\" id\u003d\"bd67be9e-bf3d-494b-9f76-aff0479ccffd\" data-root-id\u003d\"1105\"\u003e\u003c/div\u003e\n\n"
          },
          {
            "type": "HTML",
            "data": "\u003cscript\u003e (function(root) {\n  function embed_document(root) {\n    \n  var docs_json \u003d {\"6e71fb5a-11ee-4c7c-b3c0-174b3cf61f72\":{\"defs\":[],\"roots\":{\"references\":[{\"attributes\":{\"below\":[{\"id\":\"1114\"}],\"center\":[{\"id\":\"1117\"},{\"id\":\"1121\"}],\"left\":[{\"id\":\"1118\"}],\"renderers\":[{\"id\":\"1139\"},{\"id\":\"1144\"}],\"title\":{\"id\":\"1158\"},\"toolbar\":{\"id\":\"1129\"},\"x_range\":{\"id\":\"1106\"},\"x_scale\":{\"id\":\"1110\"},\"y_range\":{\"id\":\"1108\"},\"y_scale\":{\"id\":\"1112\"}},\"id\":\"1105\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"data\":{\"x\":{\"__ndarray__\":\"[encoded data, 10667 characters]\u003d\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[1000]},\"y\":{\"__ndarray__\":\"[encoded data, 10667 characters]\u003d\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[1000]}},\"selected\":{\"id\":\"1166\"},\"selection_policy\":{\"id\":\"1165\"}},\"id\":\"1136\",\"type\":\"ColumnDataSource\"},{\"attributes\":{},\"id\":\"1168\",\"type\":\"Selection\"},{\"attributes\":{\"data\":{\"x\":{\"__ndarray__\":\"[encoded data, 10667 characters]\u003d\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[1000]},\"y\":{\"__ndarray__\":\"[encoded data, 10667 characters]\u003d\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[1000]}},\"selected\":{\"id\":\"1168\"},\"selection_policy\":{\"id\":\"1167\"}},\"id\":\"1141\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"data_source\":{\"id\":\"1141\"},\"glyph\":{\"id\":\"1142\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1143\"},\"view\":{\"id\":\"1145\"}},\"id\":\"1144\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"axis\":{\"id\":\"1114\"},\"ticker\":null},\"id\":\"1117\",\"type\":\"Grid\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"red\",\"line_width\":3,\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1143\",\"type\":\"Line\"},{\"attributes\":{\"formatter\":{\"id\":\"1161\"},\"major_label_policy\":{\"id\":\"1160\"},\"ticker\":{\"id\":\"1119\"}},\"id\":\"1118\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"1115\",\"type\":\"BasicTicker\"},{\"attributes\":{\"formatter\":{\"id\":\"1164\"},\"major_label_policy\":{\"id\":\"1163\"},\"ticker\":{\"id\":\"1115\"}},\"id\":\"1114\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"1110\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1127\",\"type\":\"HelpTool\"},{\"attributes\":{},\"id\":\"1108\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1167\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"line_color\":\"red\",\"line_width\":3,\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1142\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1126\",\"type\":\"ResetTool\"},{\"attributes\":{\"axis\":{\"id\":\"1118\"},\"dimension\":1,\"ticker\":null},\"id\":\"1121\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1125\",\"type\":\"SaveTool\"},{\"attributes\":{\"fill_color\":{\"value\":\"#1f77b4\"},\"line_color\":{\"value\":\"#1f77b4\"},\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1137\",\"type\":\"Circle\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1138\",\"type\":\"Circle\"},{\"attributes\":{\"overlay\":{\"id\":\"1128\"}},\"id\":\"1124\",\"type\":\"BoxZoomTool\"},{\"attributes\":{},\"id\":\"1119\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"1161\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1112\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1123\",\"type\":\"WheelZoomTool\"},{\"attributes\":{},\"id\":\"1160\",\"type\":\"AllLabels\"},{\"attributes\":{},\"id\":\"1122\",\"type\":\"PanTool\"},{\"attributes\":{\"data_source\":{\"id\":\"1136\"},\"glyph\":{\"id\":\"1137\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1138\"},\"view\":{\"id\":\"1140\"}},\"id\":\"1139\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"bottom_units\":\"screen\",\"fill_alpha\":0.5,\"fill_color\":\"lightgrey\",\"left_units\":\"screen\",\"level\":\"overlay\",\"line_alpha\":1.0,\"line_color\":\"black\",\"line_dash\":[4,4],\"line_width\":2,\"right_units\":\"screen\",\"syncable\":false,\"top_units\":\"screen\"},\"id\":\"1128\",\"type\":\"BoxAnnotation\"},{\"attributes\":{\"source\":{\"id\":\"1136\"}},\"id\":\"1140\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"1106\",\"type\":\"DataRange1d\"},{\"attributes\":{\"source\":{\"id\":\"1141\"}},\"id\":\"1145\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"1163\",\"type\":\"AllLabels\"},{\"attributes\":{},\"id\":\"1165\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"active_multi\":null,\"tools\":[{\"id\":\"1122\"},{\"id\":\"1123\"},{\"id\":\"1124\"},{\"id\":\"1125\"},{\"id\":\"1126\"},{\"id\":\"1127\"}]},\"id\":\"1129\",\"type\":\"Toolbar\"},{\"attributes\":{},\"id\":\"1164\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1166\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1158\",\"type\":\"Title\"}],\"root_ids\":[\"1105\"]},\"title\":\"Bokeh Application\",\"version\":\"2.3.3\"}};\n  var render_items \u003d [{\"docid\":\"6e71fb5a-11ee-4c7c-b3c0-174b3cf61f72\",\"root_ids\":[\"1105\"],\"roots\":{\"1105\":\"bd67be9e-bf3d-494b-9f76-aff0479ccffd\"}}];\n  root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n\n  }\n  if (root.Bokeh !\u003d\u003d undefined) {\n    embed_document(root);\n  } else {\n    var attempts \u003d 0;\n    var timer \u003d setInterval(function(root) {\n      if (root.Bokeh !\u003d\u003d undefined) {\n        clearInterval(timer);\n        embed_document(root);\n      } else {\n        attempts++;\n        if (attempts \u003e 100) {\n          clearInterval(timer);\n          console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n        }\n      }\n    }, 10, root)\n  }\n})(window); \u003c/script\u003e\n"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {},
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1578387637679_-622761908",
      "id": "20170621-120503_149885741",
      "dateCreated": "2020-01-07 17:00:37.679",
      "dateStarted": "2021-07-26 04:32:24.308",
      "dateFinished": "2021-07-26 04:32:24.545",
      "status": "FINISHED"
    },
    {
      "title": "Loss",
      "text": "%spark.pyspark\n\np \u003d figure()\np.line(range(len(trainingSummary.objectiveHistory)), trainingSummary.objectiveHistory, color\u003d\u0027blue\u0027, line_width\u003d3)\nshow(p)\n\n\n\n",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:32:24.608",
      "progress": 0,
      "config": {
        "editorSetting": {
          "language": "python",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "colWidth": 6.0,
        "editorMode": "ace/mode/python",
        "title": true,
        "results": {},
        "enabled": true,
        "fontSize": 9.0,
        "runOnSelectionChange": true,
        "checkEmpty": true
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "HTML",
            "data": "\n\n\n\n\n\n  \u003cdiv class\u003d\"bk-root\" id\u003d\"f4804a89-bbda-45bb-b2c9-7f4348ad6b73\" data-root-id\u003d\"1235\"\u003e\u003c/div\u003e\n\n"
          },
          {
            "type": "HTML",
            "data": "\u003cscript\u003e (function(root) {\n  function embed_document(root) {\n    \n  var docs_json \u003d {\"4ef43c57-2853-4063-a190-1291d756a0ea\":{\"defs\":[],\"roots\":{\"references\":[{\"attributes\":{\"below\":[{\"id\":\"1244\"}],\"center\":[{\"id\":\"1247\"},{\"id\":\"1251\"}],\"left\":[{\"id\":\"1248\"}],\"renderers\":[{\"id\":\"1269\"}],\"title\":{\"id\":\"1296\"},\"toolbar\":{\"id\":\"1259\"},\"x_range\":{\"id\":\"1236\"},\"x_scale\":{\"id\":\"1240\"},\"y_range\":{\"id\":\"1238\"},\"y_scale\":{\"id\":\"1242\"}},\"id\":\"1235\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{},\"id\":\"1296\",\"type\":\"Title\"},{\"attributes\":{\"axis\":{\"id\":\"1244\"},\"ticker\":null},\"id\":\"1247\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1298\",\"type\":\"AllLabels\"},{\"attributes\":{},\"id\":\"1256\",\"type\":\"ResetTool\"},{\"attributes\":{\"bottom_units\":\"screen\",\"fill_alpha\":0.5,\"fill_color\":\"lightgrey\",\"left_units\":\"screen\",\"level\":\"overlay\",\"line_alpha\":1.0,\"line_color\":\"black\",\"line_dash\":[4,4],\"line_width\":2,\"right_units\":\"screen\",\"syncable\":false,\"top_units\":\"screen\"},\"id\":\"1258\",\"type\":\"BoxAnnotation\"},{\"attributes\":{\"overlay\":{\"id\":\"1258\"}},\"id\":\"1254\",\"type\":\"BoxZoomTool\"},{\"attributes\":{},\"id\":\"1302\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1255\",\"type\":\"SaveTool\"},{\"attributes\":{\"data\":{\"x\":[0,1,2],\"y\":[0.49950000000000033,0.424180453174143,0.18300563355387592]},\"selected\":{\"id\":\"1304\"},\"selection_policy\":{\"id\":\"1303\"}},\"id\":\"1266\",\"type\":\"ColumnDataSource\"},{\"attributes\":{},\"id\":\"1238\",\"type\":\"DataRange1d\"},{\"attributes\":{\"formatter\":{\"id\":\"1299\"},\"major_label_policy\":{\"id\":\"1298\"},\"ticker\":{\"id\":\"1249\"}},\"id\":\"1248\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"1236\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1242\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1252\",\"type\":\"PanTool\"},{\"attributes\":{\"axis\":{\"id\":\"1248\"},\"dimension\":1,\"ticker\":null},\"id\":\"1251\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1303\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"line_color\":\"blue\",\"line_width\":3,\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1267\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1301\",\"type\":\"AllLabels\"},{\"attributes\":{},\"id\":\"1304\",\"type\":\"Selection\"},{\"attributes\":{\"active_multi\":null,\"tools\":[{\"id\":\"1252\"},{\"id\":\"1253\"},{\"id\":\"1254\"},{\"id\":\"1255\"},{\"id\":\"1256\"},{\"id\":\"1257\"}]},\"id\":\"1259\",\"type\":\"Toolbar\"},{\"attributes\":{},\"id\":\"1257\",\"type\":\"HelpTool\"},{\"attributes\":{},\"id\":\"1253\",\"type\":\"WheelZoomTool\"},{\"attributes\":{},\"id\":\"1249\",\"type\":\"BasicTicker\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"blue\",\"line_width\":3,\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1268\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1299\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"source\":{\"id\":\"1266\"}},\"id\":\"1270\",\"type\":\"CDSView\"},{\"attributes\":{\"formatter\":{\"id\":\"1302\"},\"major_label_policy\":{\"id\":\"1301\"},\"ticker\":{\"id\":\"1245\"}},\"id\":\"1244\",\"type\":\"LinearAxis\"},{\"attributes\":{\"data_source\":{\"id\":\"1266\"},\"glyph\":{\"id\":\"1267\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1268\"},\"view\":{\"id\":\"1270\"}},\"id\":\"1269\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1245\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"1240\",\"type\":\"LinearScale\"}],\"root_ids\":[\"1235\"]},\"title\":\"Bokeh Application\",\"version\":\"2.3.3\"}};\n  var render_items \u003d [{\"docid\":\"4ef43c57-2853-4063-a190-1291d756a0ea\",\"root_ids\":[\"1235\"],\"roots\":{\"1235\":\"f4804a89-bbda-45bb-b2c9-7f4348ad6b73\"}}];\n  root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n\n  }\n  if (root.Bokeh !\u003d\u003d undefined) {\n    embed_document(root);\n  } else {\n    var attempts \u003d 0;\n    var timer \u003d setInterval(function(root) {\n      if (root.Bokeh !\u003d\u003d undefined) {\n        clearInterval(timer);\n        embed_document(root);\n      } else {\n        attempts++;\n        if (attempts \u003e 100) {\n          clearInterval(timer);\n          console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n        }\n      }\n    }, 10, root)\n  }\n})(window); \u003c/script\u003e\n"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {},
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1578387637681_1726402393",
      "id": "20170621-120529_381271006",
      "dateCreated": "2020-01-07 17:00:37.681",
      "dateStarted": "2021-07-26 04:32:24.611",
      "dateFinished": "2021-07-26 04:32:24.836",
      "status": "FINISHED"
    },
    {
      "title": "Download data for logics regression",
      "text": "%sh\n\ncd /tmp\nrm -rf sample_binary_classification_data.txt\nwget https://github.com/apache/spark/raw/master/data/mllib/sample_binary_classification_data.txt\n",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:32:24.911",
      "progress": 0,
      "config": {
        "runOnSelectionChange": true,
        "title": true,
        "checkEmpty": true,
        "colWidth": 12.0,
        "fontSize": 9.0,
        "enabled": true,
        "results": {},
        "editorSetting": {
          "language": "sh",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": false
        },
        "editorMode": "ace/mode/sh"
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "TEXT",
            "data": "--2021-07-26 04:32:26--  https://github.com/apache/spark/raw/master/data/mllib/sample_binary_classification_data.txt\nResolving github.com (github.com)... 52.74.223.119\nConnecting to github.com (github.com)|52.74.223.119|:443... connected.\nHTTP request sent, awaiting response... 302 Found\nLocation: https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_binary_classification_data.txt [following]\n--2021-07-26 04:32:27--  https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_binary_classification_data.txt\nResolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.109.133, ...\nConnecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 104736 (102K) [text/plain]\nSaving to: ‘sample_binary_classification_data.txt’\n\n     0K .......... .......... .......... .......... .......... 48% 24.0M 0s\n    50K .......... .......... .......... .......... .......... 97% 25.4M 0s\n   100K ..                                                    100% 4351G\u003d0.004s\n\n2021-07-26 04:32:27 (25.2 MB/s) - ‘sample_binary_classification_data.txt’ saved [104736/104736]\n\n"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {},
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1579425300864_833813189",
      "id": "paragraph_1579425300864_833813189",
      "dateCreated": "2020-01-19 17:15:00.864",
      "dateStarted": "2021-07-26 04:32:24.915",
      "dateFinished": "2021-07-26 04:32:27.906",
      "status": "FINISHED"
    },
    {
      "title": "Logistic Regression",
      "text": "%spark.pyspark\n\nfrom pyspark.ml.classification import LogisticRegression\nfrom pyspark.ml.classification import NaiveBayes\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import udf\nfrom pyspark.sql.types import DoubleType\n\n# Load data\ndataset \u003d spark.read.format(\"libsvm\").load(\"file:///tmp/sample_binary_classification_data.txt\")\ndataset \u003d dataset.randomSplit([0.7,0.3])\ntrainSet, testSet \u003d (dataset[0], dataset[1])\ntrainSet.cache()\ntestSet.cache()\n\nlr1 \u003d LogisticRegression(maxIter\u003d30, regParam\u003d0.1, elasticNetParam\u003d0.3)\n\n# Fit the model\nlrModel1 \u003d lr1.fit(trainSet)\nlrTrainingSummary1 \u003d lrModel1.summary\n\n# Obtain the objective per iteration\nobjectiveHistory1 \u003d lrTrainingSummary1.objectiveHistory\n\nevaluateSummary \u003d lrModel1.evaluate(testSet)\nevaluateSummary.areaUnderROC",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:32:27.917",
      "progress": 33,
      "config": {
        "editorSetting": {
          "language": "python",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "colWidth": 6.0,
        "editorMode": "ace/mode/python",
        "title": true,
        "results": {},
        "enabled": true,
        "fontSize": 9.0,
        "runOnSelectionChange": true,
        "checkEmpty": true
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "TEXT",
            "data": "1.0"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {
        "jobUrl": {
          "propertyName": "jobUrl",
          "label": "SPARK JOB",
          "tooltip": "View in Spark web UI",
          "group": "spark",
          "values": [
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d9"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d10"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d11"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d12"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d13"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d14"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d15"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d16"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d17"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d18"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d19"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d20"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d21"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d22"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d23"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d24"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d25"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d26"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d27"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d28"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d29"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d30"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d31"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d32"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d33"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d34"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d35"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d36"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d37"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d38"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d39"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d40"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d41"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d42"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d43"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d44"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d45"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d46"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d47"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d48"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d49"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d50"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d51"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d52"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d53"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d54"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d55"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d56"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d57"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d58"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d59"
            },
            {
              "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d60"
            }
          ],
          "interpreterSettingId": "spark"
        }
      },
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1578387637682_-1775952874",
      "id": "20170621-120800_296677966",
      "dateCreated": "2020-01-07 17:00:37.682",
      "dateStarted": "2021-07-26 04:32:27.920",
      "dateFinished": "2021-07-26 04:32:33.181",
      "status": "FINISHED"
    },
    {
      "title": "Loss",
      "text": "%spark.pyspark\n\np \u003d figure()\np.line(x\u003drange(len(objectiveHistory1)), y\u003dobjectiveHistory1, color\u003d\u0027blue\u0027, line_width\u003d2, legend\u003d\u0027Loss of LR1\u0027)\nshow(p)",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:32:33.237",
      "progress": 0,
      "config": {
        "editorSetting": {
          "language": "python",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "colWidth": 6.0,
        "editorMode": "ace/mode/python",
        "title": true,
        "results": {
          "0": {
            "graph": {
              "mode": "table",
              "height": 300.0,
              "optionOpen": false,
              "setting": {
                "table": {
                  "tableGridState": {},
                  "tableColumnTypeState": {
                    "names": {
                      "Col1": "string",
                      "Col2": "string"
                    },
                    "updated": false
                  },
                  "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]",
                  "tableOptionValue": {
                    "useFilter": false,
                    "showPagination": false,
                    "showAggregationFooter": false
                  },
                  "updated": false,
                  "initialized": false
                }
              },
              "commonSetting": {}
            }
          }
        },
        "enabled": true,
        "fontSize": 9.0,
        "runOnSelectionChange": true,
        "checkEmpty": true
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": [
          {
            "type": "TEXT",
            "data": "BokehDeprecationWarning: \u0027legend\u0027 keyword is deprecated, use explicit \u0027legend_label\u0027, \u0027legend_field\u0027, or \u0027legend_group\u0027 keywords instead\n\n"
          },
          {
            "type": "HTML",
            "data": "\n\n\n\n\n\n  \u003cdiv class\u003d\"bk-root\" id\u003d\"fbf995d8-b3f7-4956-8e2c-f38b3c9739c2\" data-root-id\u003d\"1361\"\u003e\u003c/div\u003e\n\n"
          },
          {
            "type": "HTML",
            "data": "\u003cscript\u003e (function(root) {\n  function embed_document(root) {\n    \n  var docs_json \u003d {\"fbe1a61a-449e-4d9a-9746-6144a366b2fc\":{\"defs\":[],\"roots\":{\"references\":[{\"attributes\":{\"below\":[{\"id\":\"1370\"}],\"center\":[{\"id\":\"1373\"},{\"id\":\"1377\"},{\"id\":\"1408\"}],\"left\":[{\"id\":\"1374\"}],\"renderers\":[{\"id\":\"1395\"}],\"title\":{\"id\":\"1398\"},\"toolbar\":{\"id\":\"1385\"},\"x_range\":{\"id\":\"1362\"},\"x_scale\":{\"id\":\"1366\"},\"y_range\":{\"id\":\"1364\"},\"y_scale\":{\"id\":\"1368\"}},\"id\":\"1361\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{},\"id\":\"1403\",\"type\":\"AllLabels\"},{\"attributes\":{},\"id\":\"1404\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1364\",\"type\":\"DataRange1d\"},{\"attributes\":{\"active_multi\":null,\"tools\":[{\"id\":\"1378\"},{\"id\":\"1379\"},{\"id\":\"1380\"},{\"id\":\"1381\"},{\"id\":\"1382\"},{\"id\":\"1383\"}]},\"id\":\"1385\",\"type\":\"Toolbar\"},{\"attributes\":{\"formatter\":{\"id\":\"1401\"},\"major_label_policy\":{\"id\":\"1400\"},\"ticker\":{\"id\":\"1375\"}},\"id\":\"1374\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"1401\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1378\",\"type\":\"PanTool\"},{\"attributes\":{\"source\":{\"id\":\"1392\"}},\"id\":\"1396\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"1400\",\"type\":\"AllLabels\"},{\"attributes\":{\"line_color\":\"blue\",\"line_width\":2,\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1393\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1392\"},\"glyph\":{\"id\":\"1393\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1394\"},\"view\":{\"id\":\"1396\"}},\"id\":\"1395\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1383\",\"type\":\"HelpTool\"},{\"attributes\":{},\"id\":\"1375\",\"type\":\"BasicTicker\"},{\"attributes\":{\"axis\":{\"id\":\"1374\"},\"dimension\":1,\"ticker\":null},\"id\":\"1377\",\"type\":\"Grid\"},{\"attributes\":{\"data\":{\"x\":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30],\"y\":[0.6808899872666542,0.5193149105310685,0.2874985300029122,0.22923003748868925,0.20473375680826328,0.1953674220772277,0.18949809372789317,0.18435445121425975,0.1815618864948226,0.17851244239408526,0.17779897882905238,0.17756756771757415,0.17721250774034747,0.17707600841747947,0.17694480114422378,0.1768690936945488,0.17677186077112564,0.17661851948024643,0.17647283165576436,0.176255870258787,0.17612791357564886,0.17595469192239996,0.1757007501368575,0.17562495918126636,0.17557095748782373,0.17553473311536275,0.17550274930372053,0.17545065914490482,0.17541682560416572,0.1753802528156692,0.17536467284861282]},\"selected\":{\"id\":\"1406\"},\"selection_policy\":{\"id\":\"1405\"}},\"id\":\"1392\",\"type\":\"ColumnDataSource\"},{\"attributes\":{},\"id\":\"1379\",\"type\":\"WheelZoomTool\"},{\"attributes\":{},\"id\":\"1398\",\"type\":\"Title\"},{\"attributes\":{\"label\":{\"value\":\"Loss of LR1\"},\"renderers\":[{\"id\":\"1395\"}]},\"id\":\"1409\",\"type\":\"LegendItem\"},{\"attributes\":{\"overlay\":{\"id\":\"1384\"}},\"id\":\"1380\",\"type\":\"BoxZoomTool\"},{\"attributes\":{\"bottom_units\":\"screen\",\"fill_alpha\":0.5,\"fill_color\":\"lightgrey\",\"left_units\":\"screen\",\"level\":\"overlay\",\"line_alpha\":1.0,\"line_color\":\"black\",\"line_dash\":[4,4],\"line_width\":2,\"right_units\":\"screen\",\"syncable\":false,\"top_units\":\"screen\"},\"id\":\"1384\",\"type\":\"BoxAnnotation\"},{\"attributes\":{\"axis\":{\"id\":\"1370\"},\"ticker\":null},\"id\":\"1373\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1405\",\"type\":\"UnionRenderers\"},{\"attributes\":{},\"id\":\"1366\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1381\",\"type\":\"SaveTool\"},{\"attributes\":{},\"id\":\"1406\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1382\",\"type\":\"ResetTool\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"blue\",\"line_width\":2,\"x\":{\"field\":\"x\"},\"y\":{\"field\":\"y\"}},\"id\":\"1394\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1368\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1362\",\"type\":\"DataRange1d\"},{\"attributes\":{\"formatter\":{\"id\":\"1404\"},\"major_label_policy\":{\"id\":\"1403\"},\"ticker\":{\"id\":\"1371\"}},\"id\":\"1370\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"1371\",\"type\":\"BasicTicker\"},{\"attributes\":{\"items\":[{\"id\":\"1409\"}]},\"id\":\"1408\",\"type\":\"Legend\"}],\"root_ids\":[\"1361\"]},\"title\":\"Bokeh Application\",\"version\":\"2.3.3\"}};\n  var render_items \u003d [{\"docid\":\"fbe1a61a-449e-4d9a-9746-6144a366b2fc\",\"root_ids\":[\"1361\"],\"roots\":{\"1361\":\"fbf995d8-b3f7-4956-8e2c-f38b3c9739c2\"}}];\n  root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n\n  }\n  if (root.Bokeh !\u003d\u003d undefined) {\n    embed_document(root);\n  } else {\n    var attempts \u003d 0;\n    var timer \u003d setInterval(function(root) {\n      if (root.Bokeh !\u003d\u003d undefined) {\n        clearInterval(timer);\n        embed_document(root);\n      } else {\n        attempts++;\n        if (attempts \u003e 100) {\n          clearInterval(timer);\n          console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n        }\n      }\n    }, 10, root)\n  }\n})(window); \u003c/script\u003e\n"
          }
        ]
      },
      "apps": [],
      "runtimeInfos": {},
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1578387637682_-1012332713",
      "id": "20170621-121651_569823333",
      "dateCreated": "2020-01-07 17:00:37.682",
      "dateStarted": "2021-07-26 04:32:33.242",
      "dateFinished": "2021-07-26 04:32:33.574",
      "status": "FINISHED"
    },
    {
      "title": "",
      "text": "%spark.pyspark\n",
      "user": "anonymous",
      "dateUpdated": "2021-07-26 04:32:33.642",
      "progress": 0,
      "config": {
        "colWidth": 12.0,
        "fontSize": 9.0,
        "enabled": true,
        "results": {},
        "editorSetting": {
          "language": "python",
          "editOnDblClick": false,
          "completionKey": "TAB",
          "completionSupport": true
        },
        "editorMode": "ace/mode/python"
      },
      "settings": {
        "params": {},
        "forms": {}
      },
      "results": {
        "code": "SUCCESS",
        "msg": []
      },
      "apps": [],
      "runtimeInfos": {},
      "progressUpdateIntervalMs": 500,
      "jobName": "paragraph_1578387637682_-181840828",
      "id": "20170621-155011_1790753917",
      "dateCreated": "2020-01-07 17:00:37.682",
      "dateStarted": "2021-07-26 04:32:33.645",
      "dateFinished": "2021-07-26 04:32:33.866",
      "status": "FINISHED"
    }
  ],
  "name": "4. Spark MlLib",
  "id": "2EZFM3GJA",
  "defaultInterpreterGroup": "spark",
  "version": "0.9.0-SNAPSHOT",
  "noteParams": {},
  "noteForms": {},
  "angularObjects": {},
  "config": {
    "isZeppelinNotebookCronEnable": false
  },
  "info": {
    "isRunning": true
  }
}