[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

classic Classic list List threaded Threaded
161 messages Options
1234 ... 9
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
GitHub user lstav opened a pull request:

    https://github.com/apache/accumulo/pull/242

    ACCUMULO-2181/3005 REST API and new Monitor UI

    Based on the original code from @joshelser, added a REST API to the Monitor to access all the information from the current Monitor, modernized the Monitor with a new UI using Bootstrap, moved all server functionality to the client (such as sorting tables), and other improvements.
   
    Most of the code to serve the REST server and resources server was done by @ctubbsii.
   
    Some of the back end stuff still need some polishing such as the JS and CSS includes which are using CDNs, which @ctubbsii will look into adding it to the packaging process, and Maven related nuances (@ctubbsii will also help with that).
   
   
    New features/old features that have been modernized include:
    1. A new notification functionality for the navbar. The user will see a summary of all logs/problems as well as level of logs from the navbar. The user will also see a summary of the server status in the navbar (Green, Yellow, Red).
    2. The Instance name is now part of the navbar and redirects to the "Overview page".
    3. Instance information has been moved to an "About" modal accessible from the navbar.
    4. New Namespace Filter (improved from earlier 2.0.0) using the Select2 library.
    5. New calls to delete logs and problems without redirecting the page.
    6. New Auto-Refresh implementation that uses Ajax to refresh the information.
   
    Any feedback on the code and the new UI will be appreciated.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lstav/accumulo ACCUMULO-MONITOR

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/accumulo/pull/242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #242
   
----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo issue #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
Github user mikewalch commented on the issue:

    https://github.com/apache/accumulo/pull/242
 
    I started reviewing this.  I was able to build a tarball and run Accumulo locally with the new monitor.  However, it looks like the integration tests are failing due to missing/extra dependencies in the monitor pom.xml.  I will create a PR to your branch to help you fix up those dependencies and get the tests working again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo issue #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user lstav commented on the issue:

    https://github.com/apache/accumulo/pull/242
 
    @mikewalch thanks, I'll merge the PR when it is available.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user mikewalch commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110012789
 
    --- Diff: server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java ---
    @@ -461,25 +452,9 @@ public void run(String hostname) {
           try {
             log.debug("Creating monitor on port " + port);
             server = new EmbeddedWebServer(hostname, port);
    -        server.addServlet(DefaultServlet.class, "/");
    -        server.addServlet(OperationServlet.class, "/op");
    -        server.addServlet(MasterServlet.class, "/master");
    -        server.addServlet(TablesServlet.class, "/tables");
    -        server.addServlet(TServersServlet.class, "/tservers");
    -        server.addServlet(ProblemServlet.class, "/problems");
    -        server.addServlet(GcStatusServlet.class, "/gc");
    -        server.addServlet(LogServlet.class, "/log");
    -        server.addServlet(XMLServlet.class, "/xml");
    -        server.addServlet(JSONServlet.class, "/json");
    -        server.addServlet(VisServlet.class, "/vis");
    -        server.addServlet(ScanServlet.class, "/scans");
    -        server.addServlet(BulkImportServlet.class, "/bulkImports");
    -        server.addServlet(Summary.class, "/trace/summary");
    -        server.addServlet(ListType.class, "/trace/listType");
    -        server.addServlet(ShowTrace.class, "/trace/show");
    -        server.addServlet(ReplicationServlet.class, "/replication");
    -        if (server.isUsingSsl())
    -          server.addServlet(ShellServlet.class, "/shell");
    +        server.addServlet(getDefaultServlet(), "/resources/*");
    +        server.addServlet(getRestServlet(), "/rest/*");
    --- End diff --
   
    Nice reduction in servlets


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user mikewalch commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110013733
 
    --- Diff: server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/BasicServlet.java ---
    @@ -1,285 +0,0 @@
    -/*
    - * Licensed to the Apache Software Foundation (ASF) under one or more
    - * contributor license agreements.  See the NOTICE file distributed with
    - * this work for additional information regarding copyright ownership.
    - * The ASF licenses this file to You under the Apache License, Version 2.0
    - * (the "License"); you may not use this file except in compliance with
    - * the License.  You may obtain a copy of the License at
    - *
    - *     http://www.apache.org/licenses/LICENSE-2.0
    - *
    - * Unless required by applicable law or agreed to in writing, software
    - * distributed under the License is distributed on an "AS IS" BASIS,
    - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    - * See the License for the specific language governing permissions and
    - * limitations under the License.
    - */
    -package org.apache.accumulo.monitor.servlets;
    -
    -import static java.nio.charset.StandardCharsets.UTF_8;
    -
    -import java.io.IOException;
    -import java.io.PrintWriter;
    -import java.io.StringWriter;
    -import java.io.UnsupportedEncodingException;
    -import java.net.HttpURLConnection;
    -import java.net.URLDecoder;
    -import java.net.URLEncoder;
    -import java.util.Date;
    -import java.util.List;
    -
    -import javax.servlet.ServletException;
    -import javax.servlet.http.Cookie;
    -import javax.servlet.http.HttpServlet;
    -import javax.servlet.http.HttpServletRequest;
    -import javax.servlet.http.HttpServletResponse;
    -
    -import org.apache.accumulo.core.Constants;
    -import org.apache.accumulo.core.conf.Property;
    -import org.apache.accumulo.monitor.Monitor;
    -import org.apache.accumulo.server.monitor.DedupedLogEvent;
    -import org.apache.accumulo.server.monitor.LogService;
    -import org.apache.log4j.Level;
    -import org.apache.log4j.Logger;
    -
    -abstract public class BasicServlet extends HttpServlet {
    -  public static final String STANDBY_MONITOR_MESSAGE = "This is not the active Monitor";
    -
    -  private static final long serialVersionUID = 1L;
    -  protected static final Logger log = Logger.getLogger(BasicServlet.class);
    -  private String bannerText;
    -  private String bannerColor;
    -  private String bannerBackground;
    -
    -  abstract protected String getTitle(HttpServletRequest req);
    -
    -  public boolean isActiveMonitor() {
    -    // If the HighlyAvailableService is not initialized or it's not the active service, throw an exception
    -    // to prevent processing of the servlet.
    -    if (null == Monitor.HA_SERVICE_INSTANCE || !Monitor.HA_SERVICE_INSTANCE.isActiveService()) {
    -      return false;
    -    }
    -    return true;
    -  }
    -
    -  @Override
    -  public void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
    -    // Verify that this is the active Monitor instance
    -    if (!isActiveMonitor()) {
    -      resp.sendError(HttpURLConnection.HTTP_UNAVAILABLE, STANDBY_MONITOR_MESSAGE);
    -      return;
    -    }
    -    StringBuilder sb = new StringBuilder();
    -    try {
    -      Monitor.fetchData();
    -      bannerText = sanitize(Monitor.getContext().getConfiguration().get(Property.MONITOR_BANNER_TEXT));
    --- End diff --
   
    The three monitor banner configuration properties in this file should be deprecated in the `Property` class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user mikewalch commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110015222
 
    --- Diff: server/monitor/src/main/resources/resources/vis.js ---
    @@ -0,0 +1,506 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    --- End diff --
   
    There is a `vis.js.old` file that should be removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user mikewalch commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110014843
 
    --- Diff: server/monitor/src/main/resources/templates/index.ftl ---
    @@ -0,0 +1,72 @@
    +<!--
    +  Licensed to the Apache Software Foundation (ASF) under one or more
    +  contributor license agreements.  See the NOTICE file distributed with
    +  this work for additional information regarding copyright ownership.
    +  The ASF licenses this file to You under the Apache License, Version 2.0
    +  (the "License"); you may not use this file except in compliance with
    +  the License.  You may obtain a copy of the License at
    +
    +    http://www.apache.org/licenses/LICENSE-2.0
    +
    +  Unless required by applicable law or agreed to in writing, software
    +  distributed under the License is distributed on an "AS IS" BASIS,
    +  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  See the License for the specific language governing permissions and
    +  limitations under the License.
    +-->
    +<html>
    +  <head>
    +    <title>${title} - Accumulo ${version}</title>
    +    <meta http-equiv="Content-Type" content="test/html" />
    +    <meta http-equiv="Content-Script-Type" content="text/javascript" />
    +    <meta http-equiv="Content-Style-Type" content="text/css" />
    +    <link rel="shortcut icon" type="image/jng" href="/resources/favicon.png" />
    +    <script src="/resources/global.js" type="text/javascript"></script>
    +    <script src="/resources/functions.js" type="text/javascript"></script>
    +    
    +    <link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">
    +    <script src="https://code.jquery.com/jquery-1.12.4.js"></script>
    +    
    +    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
    +    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
    +    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous">
    +    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
    +
    +    <script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>
    +    <script language="javascript" type="text/javascript" src="/resources/flot/jquery.flot.js"></script>
    +    <script language="javascript" type="text/javascript" src="/resources/flot/jquery.flot.time.js"></script>
    +    <link href="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.3/css/select2.min.css" rel="stylesheet" />
    +    <script src="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.3/js/select2.min.js"></script>
    +    
    +    <link rel="stylesheet" type="text/css" href="/resources/screen.css" media="screen" />
    +    <script>
    +      /**
    +       * Sets up autorefresh on initial load
    +       */
    +      $(document).ready(function() {
    +        setupAutoRefresh();
    +      });
    +    </script>
    +    <#if js??>
    +      <script src="/resources/${js}"></script>
    +    </#if>
    +    <script src="/resources/sidebar.js"></script>
    +  </head>
    +
    +  <body>
    +    <#include "/templates/modals.ftl">
    +    <div id="content-wrapper">
    +      <div id="content">
    +        <div id="sidebar" class="navbar navbar-inverse navbar-fixed-top">
    +          <!--<#include "/templates/header.ftl">-->
    +          <#include "/templates/sidebar.ftl">
    --- End diff --
   
    `sidebar.ftl` could be renamed to something like `navbar.ftl` as its not one side.  also `header.ftl` is commented out.  Can the comment and the file be removed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user mikewalch commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110019365
 
    --- Diff: server/monitor/src/main/resources/templates/overview.ftl ---
    @@ -0,0 +1,90 @@
    +<#--
    +  Licensed to the Apache Software Foundation (ASF) under one or more
    +  contributor license agreements.  See the NOTICE file distributed with
    +  this work for additional information regarding copyright ownership.
    +  The ASF licenses this file to You under the Apache License, Version 2.0
    +  (the "License"); you may not use this file except in compliance with
    +  the License.  You may obtain a copy of the License at
    +
    +    http://www.apache.org/licenses/LICENSE-2.0
    +
    +  Unless required by applicable law or agreed to in writing, software
    +  distributed under the License is distributed on an "AS IS" BASIS,
    +  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  See the License for the specific language governing permissions and
    +  limitations under the License.
    +-->
    +      <div><h3>${title}</h3></div>
    +      <br>
    +      <div class="center-block">
    +        <table class="overview-table">
    --- End diff --
   
    It would be really nice if this page and others used [bootstrap's grid system](http://getbootstrap.com/css/#grid) (rather than `<table>` tags to structure the page).  If they did, the charts and tables would collapse into a single column as screen size of the browser shrinks.  You can test this with a desktop browser.  Check out the Accumulo website for an example.  The bootstrap code is very simple.  Something like below:
   
    ```html
    <div class="container">
      <div class="row">
        <div class="col-md-6">Accumulo master table html</div>
        <div class="col-md-6">Zookeeper table html</div>
      </div>
      <div class="row">
        <div class="col-md-6">Chart 1 html</div>
        <div class="col-md-6">Chart 2 html</div>
      </div>
    </div>
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user lstav commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110050764
 
    --- Diff: server/monitor/src/main/resources/templates/index.ftl ---
    @@ -0,0 +1,72 @@
    +<!--
    +  Licensed to the Apache Software Foundation (ASF) under one or more
    +  contributor license agreements.  See the NOTICE file distributed with
    +  this work for additional information regarding copyright ownership.
    +  The ASF licenses this file to You under the Apache License, Version 2.0
    +  (the "License"); you may not use this file except in compliance with
    +  the License.  You may obtain a copy of the License at
    +
    +    http://www.apache.org/licenses/LICENSE-2.0
    +
    +  Unless required by applicable law or agreed to in writing, software
    +  distributed under the License is distributed on an "AS IS" BASIS,
    +  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  See the License for the specific language governing permissions and
    +  limitations under the License.
    +-->
    +<html>
    +  <head>
    +    <title>${title} - Accumulo ${version}</title>
    +    <meta http-equiv="Content-Type" content="test/html" />
    +    <meta http-equiv="Content-Script-Type" content="text/javascript" />
    +    <meta http-equiv="Content-Style-Type" content="text/css" />
    +    <link rel="shortcut icon" type="image/jng" href="/resources/favicon.png" />
    +    <script src="/resources/global.js" type="text/javascript"></script>
    +    <script src="/resources/functions.js" type="text/javascript"></script>
    +    
    +    <link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">
    +    <script src="https://code.jquery.com/jquery-1.12.4.js"></script>
    +    
    +    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
    +    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
    +    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous">
    +    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
    +
    +    <script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>
    +    <script language="javascript" type="text/javascript" src="/resources/flot/jquery.flot.js"></script>
    +    <script language="javascript" type="text/javascript" src="/resources/flot/jquery.flot.time.js"></script>
    +    <link href="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.3/css/select2.min.css" rel="stylesheet" />
    +    <script src="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.3/js/select2.min.js"></script>
    +    
    +    <link rel="stylesheet" type="text/css" href="/resources/screen.css" media="screen" />
    +    <script>
    +      /**
    +       * Sets up autorefresh on initial load
    +       */
    +      $(document).ready(function() {
    +        setupAutoRefresh();
    +      });
    +    </script>
    +    <#if js??>
    +      <script src="/resources/${js}"></script>
    +    </#if>
    +    <script src="/resources/sidebar.js"></script>
    +  </head>
    +
    +  <body>
    +    <#include "/templates/modals.ftl">
    +    <div id="content-wrapper">
    +      <div id="content">
    +        <div id="sidebar" class="navbar navbar-inverse navbar-fixed-top">
    +          <!--<#include "/templates/header.ftl">-->
    +          <#include "/templates/sidebar.ftl">
    --- End diff --
   
    Thanks, initially it was a sidebar that turned into a navbar and I forgot to rename it. Yes, the comment and the file can be removed. Will do this tomorrow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user lstav commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110050790
 
    --- Diff: server/monitor/src/main/resources/resources/vis.js ---
    @@ -0,0 +1,506 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    --- End diff --
   
    Thanks, forgot to remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user lstav commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110050871
 
    --- Diff: server/monitor/src/main/resources/templates/overview.ftl ---
    @@ -0,0 +1,90 @@
    +<#--
    +  Licensed to the Apache Software Foundation (ASF) under one or more
    +  contributor license agreements.  See the NOTICE file distributed with
    +  this work for additional information regarding copyright ownership.
    +  The ASF licenses this file to You under the Apache License, Version 2.0
    +  (the "License"); you may not use this file except in compliance with
    +  the License.  You may obtain a copy of the License at
    +
    +    http://www.apache.org/licenses/LICENSE-2.0
    +
    +  Unless required by applicable law or agreed to in writing, software
    +  distributed under the License is distributed on an "AS IS" BASIS,
    +  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  See the License for the specific language governing permissions and
    +  limitations under the License.
    +-->
    +      <div><h3>${title}</h3></div>
    +      <br>
    +      <div class="center-block">
    +        <table class="overview-table">
    --- End diff --
   
    Nice suggestion! I'll look into it tomorrow and see if I can implement it with no major problems.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110076794
 
    --- Diff: server/monitor/src/main/resources/resources/functions.js ---
    @@ -0,0 +1,686 @@
    +/*
    +* Licensed to the Apache Software Foundation (ASF) under one or more
    +* contributor license agreements.  See the NOTICE file distributed with
    +* this work for additional information regarding copyright ownership.
    +* The ASF licenses this file to You under the Apache License, Version 2.0
    +* (the "License"); you may not use this file except in compliance with
    +* the License.  You may obtain a copy of the License at
    +*
    +*     http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing, software
    +* distributed under the License is distributed on an "AS IS" BASIS,
    +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +* See the License for the specific language governing permissions and
    +* limitations under the License.
    +*/
    +
    +// Suffixes for quantity
    +var QUANTITY_SUFFIX = ['', 'K', 'M', 'B', 'T', 'e15', 'e18', 'e21'];
    +// Suffixes for size
    +var SIZE_SUFFIX = ['', 'K', 'M', 'G', 'T', 'P', 'E', 'Z'];
    +
    +/**
    + * Initializes Auto Refresh to false if it is not set,
    + * and creates listeners for auto refresh
    + */
    +function setupAutoRefresh() {
    +  // Sets auto refresh to true or false
    +  if (!sessionStorage.autoRefresh) {
    +    sessionStorage.autoRefresh = 'false';
    +  }
    +  // Need this to set the initial value for the autorefresh on page load
    +  if (sessionStorage.autoRefresh == 'false') {
    +    $('.auto-refresh').parent().removeClass('active');
    +  } else {
    +    $('.auto-refresh').parent().addClass('active');
    +  }
    +  // Initializes the auto refresh on click listener
    +  $('.auto-refresh').click(function(e) {
    +    if ($(this).parent().attr('class') == 'active') {
    +      $(this).parent().removeClass('active');
    +      sessionStorage.autoRefresh = 'false';
    +    } else {
    +      $(this).parent().addClass('active');
    +      sessionStorage.autoRefresh = 'true';
    +    }
    +  });
    +}
    +
    +/**
    + * Global timer that checks for auto refresh status every 5 seconds
    + */
    +TIMER = setInterval(function() {
    +  if (sessionStorage.autoRefresh == 'true') {
    +    $('.auto-refresh').parent().addClass('active');
    +    refresh();
    +    refreshNavBar();
    +  } else {
    +    $('.auto-refresh').parent().removeClass('active');
    +  }
    +}, 5000);
    +
    +/**
    + * Empty function in case there is no refresh implementation
    + */
    +function refresh() {
    +}
    +
    +/**
    + * Converts a number to a size with suffix
    + *
    + * @param {number} size Number to convert
    + * @return {string} Number with suffix added
    + */
    +function bigNumberForSize(size) {
    +  if (size === null)
    +    size = 0;
    +  return bigNumber(size, SIZE_SUFFIX, 1024);
    +}
    +
    +/**
    + * Converts a number to a quantity with suffix
    + *
    + * @param {number} quantity Number to convert
    + * @return {string} Number with suffix added
    + */
    +function bigNumberForQuantity(quantity) {
    +  if (quantity === null)
    +    quantity = 0;
    +  return bigNumber(quantity, QUANTITY_SUFFIX, 1000);
    +}
    +
    +/**
    + * Adds the suffix to the number, converts the number to one close to the base
    + *
    + * @param {number} big Number to convert
    + * @param {array} suffixes Suffixes to use for convertion
    + * @param {number} base Base to use for convertion
    + * @return {string} The new value with the suffix
    + */
    +function bigNumber(big, suffixes, base) {
    +  // If the number is smaller than the base, return thee number with no suffix
    +  if (big < base) {
    +    return big + suffixes[0];
    +  }
    +  // Finds which suffix to use
    +  var exp = Math.floor(Math.log(big) / Math.log(base));
    +  // Divides the bumber by the equivalent suffix number
    +  var val = big / Math.pow(base, exp);
    +  // Keeps the number to 2 decimal places and adds the suffix
    +  return val.toFixed(2) + suffixes[exp];
    +}
    +
    +/**
    + * Converts the time to short number and adds unit
    + *
    + * @param {number} time Time in microseconds
    + * @return {string} The time with units
    + */
    +function timeDuration(time) {
    +  var ms, sec, min, hr, day, yr;
    +  ms = sec = min = hr = day = yr = -1;
    +
    +  time = Math.floor(time);
    +
    +  // If time is 0 return a dash
    +  if (time == 0) {
    +    return '&mdash;';
    +  }
    +
    +  // Obtains the milliseconds, if time is 0, return milliseconds, and units
    +  ms = time % 1000;
    +  time = Math.floor(time / 1000);
    +  if (time == 0) {
    +    return ms + 'ms';
    +  }
    +
    +  // Obtains the seconds, if time is 0, return seconds, milliseconds, and units
    +  sec = time % 60;
    +  time = Math.floor(time / 60);
    +  if (time == 0) {
    +    return sec + 's' + '&nbsp;' + ms + 'ms';
    +  }
    +
    +  // Obtains the minutes, if time is 0, return minutes, seconds, and units
    +  min = time % 60;
    +  time = Math.floor(time / 60);
    +  if (time == 0) {
    +    return min + 'm' + '&nbsp;' + sec + 's';
    --- End diff --
   
    tsk, mixing backend work with presentation logic! ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110074670
 
    --- Diff: server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java ---
    @@ -778,77 +777,51 @@ public static long getStartTime() {
       }
     
       public static List<Pair<Long,Double>> getLoadOverTime() {
    -    synchronized (loadOverTime) {
    -      return new ArrayList<>(loadOverTime);
    -    }
    +    return new ArrayList<>(loadOverTime);
    --- End diff --
   
    Nice simplification on these.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110075995
 
    --- Diff: server/monitor/src/main/java/org/apache/accumulo/monitor/rest/tables/TablesResource.java ---
    @@ -0,0 +1,232 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.accumulo.monitor.rest.tables;
    +
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Map.Entry;
    +import java.util.SortedMap;
    +import java.util.TreeMap;
    +import java.util.TreeSet;
    +import java.util.stream.Collectors;
    +
    +import javax.ws.rs.GET;
    +import javax.ws.rs.Path;
    +import javax.ws.rs.PathParam;
    +import javax.ws.rs.Produces;
    +import javax.ws.rs.core.MediaType;
    +
    +import org.apache.accumulo.core.client.Instance;
    +import org.apache.accumulo.core.client.impl.Namespaces;
    +import org.apache.accumulo.core.client.impl.Tables;
    +import org.apache.accumulo.core.data.Range;
    +import org.apache.accumulo.core.data.impl.KeyExtent;
    +import org.apache.accumulo.core.master.thrift.TableInfo;
    +import org.apache.accumulo.core.master.thrift.TabletServerStatus;
    +import org.apache.accumulo.core.metadata.MetadataTable;
    +import org.apache.accumulo.core.metadata.RootTable;
    +import org.apache.accumulo.monitor.Monitor;
    +import org.apache.accumulo.monitor.rest.tservers.TabletServer;
    +import org.apache.accumulo.monitor.rest.tservers.TabletServers;
    +import org.apache.accumulo.server.client.HdfsZooInstance;
    +import org.apache.accumulo.server.master.state.MetaDataTableScanner;
    +import org.apache.accumulo.server.master.state.TabletLocationState;
    +import org.apache.accumulo.server.tables.TableManager;
    +import org.apache.accumulo.server.util.TableInfoUtil;
    +import org.apache.hadoop.io.Text;
    +
    +/**
    + *
    + * Generates a tables list from the Monitor as a JSON object
    + *
    + * @since 2.0.0
    + *
    + */
    +@Path("/tables")
    +@Produces({MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML})
    +public class TablesResource {
    +
    +  private static final TabletServerStatus NO_STATUS = new TabletServerStatus();
    +
    +  /**
    +   * Generates a table list based on the namespace
    +   *
    +   * @param namespace
    +   *          Namespace used to filter the tables
    +   * @return Table list
    +   */
    +  private TablesList generateTables(String namespace) {
    +    Instance inst = Monitor.getContext().getInstance();
    +    Map<String,String> tidToNameMap = Tables.getIdToNameMap(inst);
    +    SortedMap<String,TableInfo> tableStats = new TreeMap<>();
    +
    +    if (Monitor.getMmi() != null && Monitor.getMmi().tableMap != null)
    +      for (Entry<String,TableInfo> te : Monitor.getMmi().tableMap.entrySet())
    +        tableStats.put(Tables.getPrintableTableNameFromId(tidToNameMap, te.getKey()), te.getValue());
    +
    +    Map<String,Double> compactingByTable = TableInfoUtil.summarizeTableStats(Monitor.getMmi());
    +    TableManager tableManager = TableManager.getInstance();
    +
    +    SortedMap<String,String> namespaces = Namespaces.getNameToIdMap(Monitor.getContext().getInstance());
    +
    +    TablesList tableNamespace = new TablesList();
    +    List<TableInformation> tables = new ArrayList<>();
    +
    +    // Add the tables that have the selected namespace
    +    for (String key : namespaces.keySet()) {
    +      if (namespace.equals("*") || namespace.equals(key) || (key.equals("") && namespace.equals("-"))) {
    --- End diff --
   
    Make the asterisk a static-final constant if it has special meaning.
   
    What special meaning does the hyphen have? (comment)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110076356
 
    --- Diff: server/monitor/src/main/java/org/apache/accumulo/monitor/rest/tservers/TabletServerResource.java ---
    @@ -0,0 +1,336 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.accumulo.monitor.rest.tservers;
    +
    +import java.lang.management.ManagementFactory;
    +import java.security.MessageDigest;
    +import java.util.ArrayList;
    +import java.util.Base64;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import javax.ws.rs.Consumes;
    +import javax.ws.rs.GET;
    +import javax.ws.rs.POST;
    +import javax.ws.rs.Path;
    +import javax.ws.rs.PathParam;
    +import javax.ws.rs.Produces;
    +import javax.ws.rs.QueryParam;
    +import javax.ws.rs.WebApplicationException;
    +import javax.ws.rs.core.MediaType;
    +import javax.ws.rs.core.Response.Status;
    +
    +import org.apache.accumulo.core.Constants;
    +import org.apache.accumulo.core.client.impl.ClientContext;
    +import org.apache.accumulo.core.client.impl.Tables;
    +import org.apache.accumulo.core.conf.Property;
    +import org.apache.accumulo.core.data.impl.KeyExtent;
    +import org.apache.accumulo.core.master.thrift.MasterMonitorInfo;
    +import org.apache.accumulo.core.master.thrift.RecoveryStatus;
    +import org.apache.accumulo.core.master.thrift.TabletServerStatus;
    +import org.apache.accumulo.core.rpc.ThriftUtil;
    +import org.apache.accumulo.core.tabletserver.thrift.ActionStats;
    +import org.apache.accumulo.core.tabletserver.thrift.TabletClientService;
    +import org.apache.accumulo.core.tabletserver.thrift.TabletStats;
    +import org.apache.accumulo.core.trace.Tracer;
    +import org.apache.accumulo.core.util.AddressUtil;
    +import org.apache.accumulo.core.zookeeper.ZooUtil;
    +import org.apache.accumulo.monitor.Monitor;
    +import org.apache.accumulo.monitor.rest.master.MasterResource;
    +import org.apache.accumulo.server.client.HdfsZooInstance;
    +import org.apache.accumulo.server.master.state.DeadServerList;
    +import org.apache.accumulo.server.util.ActionStatsUpdator;
    +
    +import com.google.common.net.HostAndPort;
    +
    +/**
    + *
    + * Generates tserver lists as JSON objects
    + *
    + * @since 2.0.0
    + *
    + */
    +@Path("/tservers")
    +@Produces({MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML})
    +public class TabletServerResource {
    +
    +  // Variable names become JSON keys
    +  private TabletStats total, historical;
    +
    +  /**
    +   * Generates tserver summary
    +   *
    +   * @return tserver summary
    +   */
    +  @GET
    +  public TabletServers getTserverSummary() {
    +    MasterMonitorInfo mmi = Monitor.getMmi();
    +    if (null == mmi) {
    +      throw new WebApplicationException(Status.INTERNAL_SERVER_ERROR);
    +    }
    +
    +    TabletServers tserverInfo = new TabletServers(mmi.tServerInfo.size());
    +    for (TabletServerStatus status : mmi.tServerInfo) {
    +      tserverInfo.addTablet(new TabletServer(status));
    +    }
    +
    +    tserverInfo.addBadTabletServer(new MasterResource().getTables());
    +
    +    return tserverInfo;
    +  }
    +
    +  /**
    +   * REST call to clear dead servers from list
    +   *
    +   * @param server
    +   *          Dead server to clear
    +   */
    +  @POST
    +  @Consumes(MediaType.TEXT_PLAIN)
    +  public void clearDeadServer(@QueryParam("server") String server) throws Exception {
    +    DeadServerList obit = new DeadServerList(ZooUtil.getRoot(Monitor.getContext().getInstance()) + Constants.ZDEADTSERVERS);
    +    obit.delete(server);
    +  }
    +
    +  /**
    +   * Generates a recovery tserver list
    +   *
    +   * @return Recovery tserver list
    +   */
    +  @Path("recovery")
    +  @GET
    +  public Map<String,List<Map<String,String>>> getTserverRecovery() {
    +
    +    Map<String,List<Map<String,String>>> jsonObj = new HashMap<String,List<Map<String,String>>>();
    +    List<Map<String,String>> recoveryList = new ArrayList<>();
    +    Map<String,String> recoveryObj = new HashMap<String,String>();
    +
    +    MasterMonitorInfo mmi = Monitor.getMmi();
    +    if (null == mmi) {
    +      throw new WebApplicationException(Status.INTERNAL_SERVER_ERROR);
    +    }
    +
    +    for (TabletServerStatus server : mmi.tServerInfo) {
    +      if (server.logSorts != null) {
    +        for (RecoveryStatus recovery : server.logSorts) {
    +          recoveryObj.put("server", AddressUtil.parseAddress(server.name, false).getHostText());
    +          recoveryObj.put("log", recovery.name);
    +          recoveryObj.put("time", Long.toString(recovery.runtime));
    +          recoveryObj.put("copySort", Double.toString(recovery.progress));
    +
    +          recoveryList.add(recoveryObj);
    +        }
    +      }
    +    }
    +
    +    jsonObj.put("recoveryList", recoveryList);
    +
    +    return jsonObj;
    +  }
    +
    +  /**
    +   * Generates details for the selected tserver
    +   *
    +   * @param tserverAddr
    +   *          TServer name
    +   * @return TServer details
    +   */
    +  @Path("{address}")
    +  @GET
    +  public TabletServerSummary getTserverDetails(@PathParam("address") String tserverAddr) throws Exception {
    +
    +    String tserverAddress = tserverAddr;
    +
    +    boolean tserverExists = false;
    +    if (tserverAddress != null && tserverAddress.isEmpty() == false) {
    +      for (TabletServerStatus ts : Monitor.getMmi().getTServerInfo()) {
    +        if (tserverAddress.equals(ts.getName())) {
    +          tserverExists = true;
    +          break;
    +        }
    +      }
    +    }
    +
    +    if (tserverAddress == null || tserverAddress.isEmpty() || tserverExists == false) {
    +
    +      return null;
    +    }
    +
    +    double totalElapsedForAll = 0;
    +    double splitStdDev = 0;
    +    double minorStdDev = 0;
    +    double minorQueueStdDev = 0;
    +    double majorStdDev = 0;
    +    double majorQueueStdDev = 0;
    +    double currentMinorAvg = 0;
    +    double currentMajorAvg = 0;
    +    double currentMinorStdDev = 0;
    +    double currentMajorStdDev = 0;
    +    total = new TabletStats(null, new ActionStats(), new ActionStats(), new ActionStats(), 0, 0, 0, 0);
    +    HostAndPort address = HostAndPort.fromString(tserverAddress);
    +    historical = new TabletStats(null, new ActionStats(), new ActionStats(), new ActionStats(), 0, 0, 0, 0);
    +    List<TabletStats> tsStats = new ArrayList<>();
    +
    +    try {
    +      ClientContext context = Monitor.getContext();
    +      TabletClientService.Client client = ThriftUtil.getClient(new TabletClientService.Client.Factory(), address, context);
    +      try {
    +        for (String tableId : Monitor.getMmi().tableMap.keySet()) {
    +          tsStats.addAll(client.getTabletStats(Tracer.traceInfo(), context.rpcCreds(), tableId));
    +        }
    +        historical = client.getHistoricalStats(Tracer.traceInfo(), context.rpcCreds());
    +      } finally {
    +        ThriftUtil.returnClient(client);
    +      }
    +    } catch (Exception e) {
    +      return null;
    +    }
    +
    +    List<CurrentOperations> currentOps = doCurrentOperations(tsStats);
    +
    +    if (total.minors.num != 0)
    +      currentMinorAvg = (long) (total.minors.elapsed / total.minors.num);
    +    if (total.minors.elapsed != 0 && total.minors.num != 0)
    +      currentMinorStdDev = stddev(total.minors.elapsed, total.minors.num, total.minors.sumDev);
    +    if (total.majors.num != 0)
    +      currentMajorAvg = total.majors.elapsed / total.majors.num;
    +    if (total.majors.elapsed != 0 && total.majors.num != 0 && total.majors.elapsed > total.majors.num)
    +      currentMajorStdDev = stddev(total.majors.elapsed, total.majors.num, total.majors.sumDev);
    +
    +    ActionStatsUpdator.update(total.minors, historical.minors);
    +    ActionStatsUpdator.update(total.majors, historical.majors);
    +    totalElapsedForAll += total.majors.elapsed + historical.splits.elapsed + total.minors.elapsed;
    +
    +    minorStdDev = stddev(total.minors.elapsed, total.minors.num, total.minors.sumDev);
    +    minorQueueStdDev = stddev(total.minors.queueTime, total.minors.num, total.minors.queueSumDev);
    +    majorStdDev = stddev(total.majors.elapsed, total.majors.num, total.majors.sumDev);
    +    majorQueueStdDev = stddev(total.majors.queueTime, total.majors.num, total.majors.queueSumDev);
    +    splitStdDev = stddev(historical.splits.num, historical.splits.elapsed, historical.splits.sumDev);
    +
    +    TabletServerDetailInformation details = doDetails(address, tsStats.size());
    +
    +    List<AllTimeTabletResults> allTime = doAllTimeResults(majorQueueStdDev, minorQueueStdDev, totalElapsedForAll, splitStdDev, majorStdDev, minorStdDev);
    +
    +    CurrentTabletResults currentRes = doCurrentTabletResults(currentMinorAvg, currentMinorStdDev, currentMajorAvg, currentMajorStdDev);
    +
    +    TabletServerSummary tserverDetails = new TabletServerSummary(details, allTime, currentRes, currentOps);
    +
    +    return tserverDetails;
    +  }
    +
    +  private static final int concurrentScans = Monitor.getContext().getConfiguration().getCount(Property.TSERV_READ_AHEAD_MAXCONCURRENT);
    --- End diff --
   
    This should be re-computed to ensure that the Monitor picks up new values for this configuration property.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110077224
 
    --- Diff: server/monitor/src/main/resources/templates/index.ftl ---
    @@ -0,0 +1,72 @@
    +<!--
    +  Licensed to the Apache Software Foundation (ASF) under one or more
    +  contributor license agreements.  See the NOTICE file distributed with
    +  this work for additional information regarding copyright ownership.
    +  The ASF licenses this file to You under the Apache License, Version 2.0
    +  (the "License"); you may not use this file except in compliance with
    +  the License.  You may obtain a copy of the License at
    +
    +    http://www.apache.org/licenses/LICENSE-2.0
    +
    +  Unless required by applicable law or agreed to in writing, software
    +  distributed under the License is distributed on an "AS IS" BASIS,
    +  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  See the License for the specific language governing permissions and
    +  limitations under the License.
    +-->
    +<html>
    +  <head>
    +    <title>${title} - Accumulo ${version}</title>
    +    <meta http-equiv="Content-Type" content="test/html" />
    +    <meta http-equiv="Content-Script-Type" content="text/javascript" />
    +    <meta http-equiv="Content-Style-Type" content="text/css" />
    +    <link rel="shortcut icon" type="image/jng" href="/resources/favicon.png" />
    +    <script src="/resources/global.js" type="text/javascript"></script>
    +    <script src="/resources/functions.js" type="text/javascript"></script>
    +    
    +    <link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">
    +    <script src="https://code.jquery.com/jquery-1.12.4.js"></script>
    +    
    +    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
    +    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
    +    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous">
    +    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
    +
    +    <script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>
    +    <script language="javascript" type="text/javascript" src="/resources/flot/jquery.flot.js"></script>
    +    <script language="javascript" type="text/javascript" src="/resources/flot/jquery.flot.time.js"></script>
    +    <link href="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.3/css/select2.min.css" rel="stylesheet" />
    +    <script src="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.3/js/select2.min.js"></script>
    +    
    +    <link rel="stylesheet" type="text/css" href="/resources/screen.css" media="screen" />
    +    <script>
    +      /**
    +       * Sets up autorefresh on initial load
    +       */
    +      $(document).ready(function() {
    +        setupAutoRefresh();
    +      });
    +    </script>
    +    <#if js??>
    --- End diff --
   
    what's this do?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110076627
 
    --- Diff: server/monitor/src/main/resources/resources/bulkImport.js ---
    @@ -0,0 +1,213 @@
    +/*
    +* Licensed to the Apache Software Foundation (ASF) under one or more
    +* contributor license agreements.  See the NOTICE file distributed with
    +* this work for additional information regarding copyright ownership.
    +* The ASF licenses this file to You under the Apache License, Version 2.0
    +* (the "License"); you may not use this file except in compliance with
    +* the License.  You may obtain a copy of the License at
    +*
    +*     http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing, software
    +* distributed under the License is distributed on an "AS IS" BASIS,
    +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +* See the License for the specific language governing permissions and
    +* limitations under the License.
    +*/
    +
    +/**
    + * Creates bulk import initial table
    + */
    +$(document).ready(function() {
    +  createBulkImportHeader();
    +  createServerBulkHeader();
    +  refreshBulkImport();
    +});
    +
    +/**
    + * Makes the REST calls, generates the tables with the new information
    + */
    +function refreshBulkImport() {
    +  $.ajaxSetup({
    +    async: false
    +  });
    +  getBulkImports();
    +  $.ajaxSetup({
    +    async: true
    +  });
    +  refreshBulkImportTable();
    +  refreshServerBulkTable();
    +}
    +
    +/**
    + * Used to redraw the page
    + */
    +function refresh() {
    +  refreshBulkImport();
    +}
    +
    +/**
    + * Generates the master bulk import status table
    + */
    +function refreshBulkImportTable() {
    +
    +  clearTable('masterBulkImportStatus');
    +
    +  /*
    +   * Get the bulk import value obtained earlier, if it doesn't exists,
    +   * create an empty array
    +   */
    +  var data = sessionStorage.bulkImports === undefined ?
    +      [] : JSON.parse(sessionStorage.bulkImports);
    +  var items = [];
    +
    +  /* If the data is empty, create an empty row, otherwise,
    +   * create the rows for the table
    +   */
    +  if (data.length === 0 || data.bulkImport.length === 0) {
    +    items.push('<td class="center" colspan="3"><i>Empty</i></td>');
    +  } else {
    +    $.each(data.bulkImport, function(key, val) {
    +      items.push('<td class="firstcell left" data-value="' + val.filename +
    +          '">' + val.filename + '</td>');
    +
    +      items.push('<td class="right" data-value="' + val.age + '">' + val.age +
    +          '</td>');
    +
    +      items.push('<td class="right" data-value="' + val.state + '">' +
    +          val.state + '</td>');
    +    });
    +  }
    +
    +  $('<tr/>', {
    +    html: items.join('')
    +  }).appendTo('#masterBulkImportStatus');
    +}
    +
    +/**
    + * Generates the bulk import status table
    + */
    +function refreshServerBulkTable() {
    +
    +  clearTable('bulkImportStatus');
    +
    +  /* Get the bulk import value obtained earlier, if it doesn't exists,
    +   * create an empty array
    +   */
    +  var data = sessionStorage.bulkImports === undefined ?
    +   [] : JSON.parse(sessionStorage.bulkImports);
    +  var items = [];
    +
    +  /* If the data is empty, create an empty row, otherwise
    +   * create the rows for the table
    +   */
    +  if (data.length === 0 || data.tabletServerBulkImport.length === 0) {
    +    items.push('<td class="center" colspan="3"><i>Empty</i></td>');
    +  } else {
    +    $.each(data.tabletServerBulkImport, function(key, val) {
    +      items.push('<td class="firstcell left" data-value="' + val.server +
    +          '"><a href="/tservers?s=' + val.server + '">' + val.server +
    +          '</a></td>');
    +
    +      items.push('<td class="right" data-value="' + val.importSize + '">' +
    +          val.importSize + '</td>');
    +
    +      items.push('<td class="right" data-value="' + val.oldestAge + '">' +
    +          (val.oldestAge > 0 ? val.oldestAge : '&mdash;') + '</td>');
    +    });
    +  }
    +
    +  $('<tr/>', {
    +    html: items.join('')
    +  }).appendTo('#bulkImportStatus');
    +}
    +
    +/**
    + * Sorts the bulkImportStatus table on the selected column
    + *
    + * @param {string} table Table ID to sort
    + * @param {number} n Column number to sort by
    + */
    +function sortTable(table, n) {
    +  var tableIDs = ['bulkImportStatus', 'masterBulkImportStatus'];
    +
    +  if (sessionStorage.tableColumnSort !== undefined &&
    +      sessionStorage.tableColumnSort == n &&
    +      sessionStorage.direction !== undefined) {
    +    direction = sessionStorage.direction === 'asc' ? 'desc' : 'asc';
    +  } else {
    +    direction = sessionStorage.direction === undefined ?
    +        'asc' : sessionStorage.direction;
    +  }
    +  sessionStorage.tableColumn = tableIDs[table];
    +  sessionStorage.tableColumnSort = n;
    +  sortTables(tableIDs[table], direction, n);
    +}
    +
    +/**
    + * Create tooltip for table column information
    + */
    +$(function() {
    +  $(document).tooltip();
    --- End diff --
   
    is there a reason this isn't in the `$(document).onReady`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110076465
 
    --- Diff: server/monitor/src/main/java/org/apache/accumulo/monitor/rest/zk/ZKInformation.java ---
    @@ -0,0 +1,50 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.accumulo.monitor.rest.zk;
    +
    +import java.util.ArrayList;
    +import java.util.List;
    +
    +/**
    + *
    + * Generates a list of Zookeeper server information
    + *
    + * @since 2.0.0
    + *
    + */
    +public class ZKInformation {
    --- End diff --
   
    Do we need this class to encapsulate this? Couldn't it just be replaced with List<ZooKeeper>?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110075565
 
    --- Diff: server/monitor/src/main/java/org/apache/accumulo/monitor/rest/problems/ProblemsResource.java ---
    @@ -0,0 +1,175 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.accumulo.monitor.rest.problems;
    +
    +import java.util.ArrayList;
    +import java.util.HashMap;
    +import java.util.Iterator;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Map.Entry;
    +
    +import javax.ws.rs.Consumes;
    +import javax.ws.rs.GET;
    +import javax.ws.rs.POST;
    +import javax.ws.rs.Path;
    +import javax.ws.rs.Produces;
    +import javax.ws.rs.QueryParam;
    +import javax.ws.rs.core.MediaType;
    +
    +import org.apache.accumulo.core.client.impl.Tables;
    +import org.apache.accumulo.monitor.Monitor;
    +import org.apache.accumulo.server.client.HdfsZooInstance;
    +import org.apache.accumulo.server.problems.ProblemReport;
    +import org.apache.accumulo.server.problems.ProblemReports;
    +import org.apache.accumulo.server.problems.ProblemType;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +/**
    + *
    + * Generates a problem summary and details as a JSON object
    + *
    + * @since 2.0.0
    + *
    + */
    +@Path("/problems")
    +@Produces({MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML})
    +public class ProblemsResource {
    +
    +  /**
    +   * Generates a list with the problem summary
    +   *
    +   * @return problem summary list
    +   */
    +  @GET
    +  @Path("summary")
    +  public Map<String,List<ProblemSummaryInformation>> getSummary() {
    +
    +    Map<String,List<ProblemSummaryInformation>> jsonObj = new HashMap<String,List<ProblemSummaryInformation>>();
    +
    +    List<ProblemSummaryInformation> problems = new ArrayList<>();
    +
    +    Map<String,String> tidToNameMap = Tables.getIdToNameMap(HdfsZooInstance.getInstance());
    +
    +    if (Monitor.getProblemException() == null) {
    +      for (Entry<String,Map<ProblemType,Integer>> entry : Monitor.getProblemSummary().entrySet()) {
    +        Integer readCount = null, writeCount = null, loadCount = null;
    +
    +        for (ProblemType pt : ProblemType.values()) {
    +          Integer pcount = entry.getValue().get(pt);
    +          if (pt.equals(ProblemType.FILE_READ)) {
    +            readCount = pcount;
    +          } else if (pt.equals(ProblemType.FILE_WRITE)) {
    +            writeCount = pcount;
    +          } else if (pt.equals(ProblemType.TABLET_LOAD)) {
    +            loadCount = pcount;
    +          }
    +        }
    +
    +        String tableName = Tables.getPrintableTableNameFromId(tidToNameMap, entry.getKey());
    +
    +        problems.add(new ProblemSummaryInformation(tableName, entry.getKey(), readCount, writeCount, loadCount));
    +      }
    +    }
    +    jsonObj.put("problemSummary", problems);
    +
    +    return jsonObj;
    +  }
    +
    +  /**
    +   * REST call to clear problem reports from a table
    +   *
    +   * @param tableID
    +   *          Table ID to clear problems
    +   */
    +  @POST
    +  @Consumes(MediaType.TEXT_PLAIN)
    +  @Path("summary")
    +  public void clearTableProblems(@QueryParam("s") String tableID) {
    +    Logger log = LoggerFactory.getLogger(Monitor.class);
    +    try {
    +      ProblemReports.getInstance(Monitor.getContext()).deleteProblemReports(tableID);
    +    } catch (Exception e) {
    +      log.error("Failed to delete problem reports for table " + tableID, e);
    +    }
    +  }
    +
    +  /**
    +   * Generates a list of the problem details as a JSON object
    +   *
    +   * @return problem details list
    +   */
    +  @GET
    +  @Path("details")
    +  public Map<String,List<ProblemDetailInformation>> getDetails() {
    +
    +    Map<String,List<ProblemDetailInformation>> jsonObj = new HashMap<String,List<ProblemDetailInformation>>();
    --- End diff --
   
    `new HashMap<>()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] accumulo pull request #242: ACCUMULO-2181/3005 REST API and new Monitor UI

ctubbsii
In reply to this post by ctubbsii
Github user joshelser commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/242#discussion_r110076106
 
    --- Diff: server/monitor/src/main/java/org/apache/accumulo/monitor/rest/tables/TablesResource.java ---
    @@ -0,0 +1,232 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.accumulo.monitor.rest.tables;
    +
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Map.Entry;
    +import java.util.SortedMap;
    +import java.util.TreeMap;
    +import java.util.TreeSet;
    +import java.util.stream.Collectors;
    +
    +import javax.ws.rs.GET;
    +import javax.ws.rs.Path;
    +import javax.ws.rs.PathParam;
    +import javax.ws.rs.Produces;
    +import javax.ws.rs.core.MediaType;
    +
    +import org.apache.accumulo.core.client.Instance;
    +import org.apache.accumulo.core.client.impl.Namespaces;
    +import org.apache.accumulo.core.client.impl.Tables;
    +import org.apache.accumulo.core.data.Range;
    +import org.apache.accumulo.core.data.impl.KeyExtent;
    +import org.apache.accumulo.core.master.thrift.TableInfo;
    +import org.apache.accumulo.core.master.thrift.TabletServerStatus;
    +import org.apache.accumulo.core.metadata.MetadataTable;
    +import org.apache.accumulo.core.metadata.RootTable;
    +import org.apache.accumulo.monitor.Monitor;
    +import org.apache.accumulo.monitor.rest.tservers.TabletServer;
    +import org.apache.accumulo.monitor.rest.tservers.TabletServers;
    +import org.apache.accumulo.server.client.HdfsZooInstance;
    +import org.apache.accumulo.server.master.state.MetaDataTableScanner;
    +import org.apache.accumulo.server.master.state.TabletLocationState;
    +import org.apache.accumulo.server.tables.TableManager;
    +import org.apache.accumulo.server.util.TableInfoUtil;
    +import org.apache.hadoop.io.Text;
    +
    +/**
    + *
    + * Generates a tables list from the Monitor as a JSON object
    + *
    + * @since 2.0.0
    + *
    + */
    +@Path("/tables")
    +@Produces({MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML})
    +public class TablesResource {
    +
    +  private static final TabletServerStatus NO_STATUS = new TabletServerStatus();
    +
    +  /**
    +   * Generates a table list based on the namespace
    +   *
    +   * @param namespace
    +   *          Namespace used to filter the tables
    +   * @return Table list
    +   */
    +  private TablesList generateTables(String namespace) {
    +    Instance inst = Monitor.getContext().getInstance();
    +    Map<String,String> tidToNameMap = Tables.getIdToNameMap(inst);
    +    SortedMap<String,TableInfo> tableStats = new TreeMap<>();
    +
    +    if (Monitor.getMmi() != null && Monitor.getMmi().tableMap != null)
    +      for (Entry<String,TableInfo> te : Monitor.getMmi().tableMap.entrySet())
    +        tableStats.put(Tables.getPrintableTableNameFromId(tidToNameMap, te.getKey()), te.getValue());
    +
    +    Map<String,Double> compactingByTable = TableInfoUtil.summarizeTableStats(Monitor.getMmi());
    +    TableManager tableManager = TableManager.getInstance();
    +
    +    SortedMap<String,String> namespaces = Namespaces.getNameToIdMap(Monitor.getContext().getInstance());
    +
    +    TablesList tableNamespace = new TablesList();
    +    List<TableInformation> tables = new ArrayList<>();
    +
    +    // Add the tables that have the selected namespace
    +    for (String key : namespaces.keySet()) {
    +      if (namespace.equals("*") || namespace.equals(key) || (key.equals("") && namespace.equals("-"))) {
    +        tableNamespace.addTable(new TableNamespace(key));
    +      }
    +    }
    +
    +    // Add tables to the list
    +    for (Entry<String,String> entry : Tables.getNameToIdMap(HdfsZooInstance.getInstance()).entrySet()) {
    +      String tableName = entry.getKey(), tableId = entry.getValue();
    +      TableInfo tableInfo = tableStats.get(tableName);
    +      if (null != tableInfo) {
    +        Double holdTime = compactingByTable.get(tableId);
    +        if (holdTime == null)
    +          holdTime = Double.valueOf(0.);
    +
    +        for (TableNamespace name : tableNamespace.tables) {
    +          // Check if table has the default namespace
    +          if (!tableName.contains(".") && name.namespace.equals("")) {
    --- End diff --
   
    `name.namespace.isEmpty()` instead of allocating a new string.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
1234 ... 9
Loading...