diff --git a/.gitignore b/.gitignore index 988ea68..cc989ce 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ /lib /classes /checkouts +/tmp pom.xml *.jar *.class diff --git a/README.md b/README.md index 8c86dbc..93b5100 100644 --- a/README.md +++ b/README.md @@ -41,18 +41,18 @@ You can import more than one repo into the same db. You can re-import later afte You can then (or during) connect to the same db URI with a peer. Or, just start the [Datomic REST service](http://docs.datomic.com/rest.html) and poke around: cd whereverYouPutDatomicFree - bin/rest 8080 free datomic:free://localhost:4334/ + bin/rest -p 8080 free datomic:free://localhost:4334/ -Browse to [localhost:8080](http://localhost:8080). You should see the `free` storage and the `git` db within it. +Browse to [localhost:8080/data/](http://localhost:8080/data/). You should see the `free` storage and the `git` db within it. The [schema diagram](https://github.com/downloads/Datomic/codeq/codeq.pdf) will help you get oriented. ## More info See the [intro blog post](http://blog.datomic.com/2012/10/codeq.html) and the [wiki](https://github.com/Datomic/codeq/wiki) - + ## License -Copyright © 2012 Metadata Partners, LLC +Copyright © 2012 Metadata Partners, LLC and Contributors. All rights reserved. Distributed under the Eclipse Public License, the same as Clojure. diff --git a/examples/clojure-and-contrib b/examples/clojure-and-contrib new file mode 100755 index 0000000..db35467 --- /dev/null +++ b/examples/clojure-and-contrib @@ -0,0 +1,79 @@ +#!/bin/bash + +cd `dirname $0`/.. +CODEQ_VERSION="0.1.0-SNAPSHOT" +CODEQ_ROOT=`pwd` + +DATOMIC_VERSION="0.8.3784" +DATOMIC_FILE="datomic-free-$DATOMIC_VERSION" +DATOMIC_URL=http://downloads.datomic.com/$DATOMIC_VERSION/$DATOMIC_FILE.zip + +BACKUP_FILE="clojure-and-contrib" +BACKUP_URL=http://codeq.s3.amazonaws.com/examples/$BACKUP_FILE.zip + +DB_URI="datomic:free://localhost:4334/clojure-and-contrib" + +RET=0 + +WORKING_DIR=tmp/examples/clojure-and-contrib +mkdir -p $WORKING_DIR +cd $WORKING_DIR +WORKING_DIR=`pwd` + +if [ ! -d "$BACKUP_FILE" ]; then + wget $BACKUP_URL + unzip $BACKUP_FILE.zip +fi + +if [ ! -d "$DATOMIC_FILE" ]; then + wget $DATOMIC_URL + unzip $DATOMIC_FILE.zip +fi + +#### Restore + +cd $DATOMIC_FILE + +# Start with a fresh database +rm -rf data log + +bin/transactor config/samples/free-transactor-template.properties & +TRANSACTOR_PID=$! + +(( RET += $? )) + +bin/datomic restore-db file:$WORKING_DIR/$BACKUP_FILE $DB_URI + +(( RET += $? )) + +pkill -P $TRANSACTOR_PID + +#### Verify + +bin/transactor config/samples/free-transactor-template.properties & +TRANSACTOR_PID=$! + +(( RET += $? )) + +cd $CODEQ_ROOT + +sleep 5 + +lein run -m datomic.codeq.examples.clojure-and-contrib $DB_URI + +(( RET += $? )) + +cd $CODEQ_ROOT +lein clean +lein uberjar + +git clone git@github.com:clojure/clojure.git $WORKING_DIR/clojure +cd $WORKING_DIR/clojure + +java -server -Xmx1g -jar $CODEQ_ROOT/target/codeq-$CODEQ_VERSION-standalone.jar $DB_URI + +(( RET += $? )) + +pkill -P $TRANSACTOR_PID + +exit $RET diff --git a/examples/src/datomic/codeq/examples/clojure_and_contrib.clj b/examples/src/datomic/codeq/examples/clojure_and_contrib.clj new file mode 100644 index 0000000..8a292e6 --- /dev/null +++ b/examples/src/datomic/codeq/examples/clojure_and_contrib.clj @@ -0,0 +1,46 @@ +(ns datomic.codeq.examples.clojure-and-contrib + (:require [datomic.api :as d :refer [q]] + [clojure.pprint :refer [pprint]])) + +(defn -main [& [database-uri]] + (assert database-uri) + (println "Running clojure-and-contrib examples with database" database-uri) + (try + (let [conn (d/connect database-uri) + db (-> conn d/db (d/as-of 435691)) + repos (map first + (q '[:find ?repo + :where [?e :repo/uri ?repo]] + db)) + namespaces (map first + (q '[:find ?ns + :where + [?e :clj/ns ?n] + [?n :code/name ?ns]] + db)) + definitions (reduce (fn [agg [o d]] + (update-in agg [o] (fnil conj []) d)) + {} + (q '[:find ?op ?def + :where + [?e :clj/def ?d] + [?e :clj/defop ?op] + [?d :code/name ?def]] + db))] + (println) + (println "#### Repos:") + (println) + (pprint repos) + (println) + (println "#### Namespaces:") + (println) + (pprint namespaces) + (println) + (println "#### Definitions:") + (println) + (pprint definitions) + (assert (= 44 (-> definitions keys count))) + (assert (= 33 (-> "defne" definitions count)))) + (finally + ;; (shutdown-agents) + (System/exit 0)))) diff --git a/project.clj b/project.clj index bc473d2..1ce950d 100644 --- a/project.clj +++ b/project.clj @@ -5,6 +5,7 @@ :url "http://www.eclipse.org/legal/epl-v10.html"} :main datomic.codeq.core :plugins [[lein-tar "1.1.0"]] - :dependencies [[com.datomic/datomic-free "0.8.3546"] + :dependencies [[com.datomic/datomic-free "0.9.4699"] [commons-codec "1.7"] - [org.clojure/clojure "1.5.0-alpha6"]]) + [org.clojure/clojure "1.5.1"]] + :source-paths ["src" "examples/src"]) diff --git a/src/datomic/codeq/analyzer.clj b/src/datomic/codeq/analyzer.clj index d82a3be..4d662eb 100644 --- a/src/datomic/codeq/analyzer.clj +++ b/src/datomic/codeq/analyzer.clj @@ -1,4 +1,4 @@ -;; Copyright (c) Metadata Partners, LLC. All rights reserved. +;; Copyright (c) Metadata Partners, LLC and Contributors. All rights reserved. ;; The use and distribution terms for this software are covered by the ;; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php) ;; which can be found in the file epl-v10.html at the root of this distribution. @@ -38,9 +38,9 @@ (-> sb str .trim))) (defn loc - "Returns zero-based [line col endline endcol] given one-based - \"line col endline endcol\" string" - [loc-string] + "Returns zero-based [line col endline endcol] given one-based + \"line col endline endcol\" string" + [loc-string] (mapv dec (read-string (str "[" loc-string "]")))) (defn line-offsets @@ -50,19 +50,19 @@ index is the length of the string." [^String s] (let [nl (long \newline)] - (persistent! + (persistent! (loop [ret (transient [0]), i 0] - (if (= i (.length s)) + (if (= i (.length s)) ret - (recur (if (= (.codePointAt s i) nl) - (conj! ret (inc i)) - ret) + (recur (if (= (.codePointAt s i) nl) + (conj! ret (inc i)) + ret) (inc i))))))) (defn segment "Given a string and line offsets, returns text from (zero-based) - line and col to endline/endcol (exclusive)" + line and col to endline/endcol (exclusive)" [^String s line-offsets line col endline endcol] - (subs s + (subs s (+ (nth line-offsets line) col) (+ (nth line-offsets endline) endcol))) diff --git a/src/datomic/codeq/analyzers/clj.clj b/src/datomic/codeq/analyzers/clj.clj index d27ff4a..491d776 100644 --- a/src/datomic/codeq/analyzers/clj.clj +++ b/src/datomic/codeq/analyzers/clj.clj @@ -1,4 +1,4 @@ -;; Copyright (c) Metadata Partners, LLC. All rights reserved. +;; Copyright (c) Metadata Partners, LLC and Contributors. All rights reserved. ;; The use and distribution terms for this software are covered by the ;; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php) ;; which can be found in the file epl-v10.html at the root of this distribution. @@ -8,7 +8,7 @@ (ns datomic.codeq.analyzers.clj (:require [datomic.api :as d] - [datomic.codeq.util :refer [cond-> index->id-fn tempid?]] + [datomic.codeq.util :refer [index->id-fn tempid?]] [datomic.codeq.analyzer :as az])) (defn analyze-1 @@ -31,9 +31,9 @@ defing (and ns (symbol? op) (.startsWith (name op) "def")) - + naming (let [nsym (second x)] - (cond + (cond ns? (str nsym) defing (if (namespace nsym) (str nsym) @@ -41,26 +41,26 @@ nameid (when naming (codename->id naming)) - ret (cond-> ret + ret (cond-> ret (tempid? codeqid) (conj {:db/id codeqid :codeq/file f :codeq/loc loc :codeq/code codeid}) - + ns? (conj [:db/add codeqid :clj/ns nameid]) defing (conj [:db/add codeqid :clj/def nameid] [:db/add codeqid :clj/defop (str op)]) - + (tempid? nameid) (conj [:db/add nameid :code/name naming]))] [ret (assoc ctx :added added)]) [ret ctx])) -(defn analyze +(defn analyze [db f src] (with-open [r (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. src))] (let [loffs (az/line-offsets src) diff --git a/src/datomic/codeq/core.clj b/src/datomic/codeq/core.clj index e39ed0c..ef83b45 100644 --- a/src/datomic/codeq/core.clj +++ b/src/datomic/codeq/core.clj @@ -1,4 +1,4 @@ -;; Copyright (c) Metadata Partners, LLC. All rights reserved. +;; Copyright (c) Metadata Partners, LLC and Contributors. All rights reserved. ;; The use and distribution terms for this software are covered by the ;; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php) ;; which can be found in the file epl-v10.html at the root of this distribution. @@ -11,7 +11,7 @@ [clojure.java.io :as io] [clojure.set] [clojure.string :as string] - [datomic.codeq.util :refer [cond-> index->id-fn tempid?]] + [datomic.codeq.util :refer [index->id-fn tempid?]] [datomic.codeq.analyzer :as az] [datomic.codeq.analyzers.clj]) (:import java.util.Date) @@ -82,7 +82,7 @@ :db/cardinality :db.cardinality/many :db/doc "Associate repo with these git commits" :db.install/_attribute :db.part/db} - + {:db/id #db/id[:db.part/db] :db/ident :repo/uri :db/valueType :db.type/string @@ -250,11 +250,11 @@ :db.install/_attribute :db.part/db} ]) -(defn ^java.io.Reader exec-stream +(defn ^java.io.Reader exec-stream [^String cmd] (-> (Runtime/getRuntime) - (.exec cmd) - .getInputStream + (.exec cmd) + .getInputStream io/reader)) (defn ensure-schema [conn] @@ -303,8 +303,8 @@ noff (.lastIndexOf uri "/") noff (if (not (pos? noff)) (.lastIndexOf uri ":") noff) name (subs uri (inc noff)) - _ (assert (and (pos? (count name)) (.endsWith name ".git")) "Can't find remote origin") - name (subs name 0 (.indexOf name "."))] + _ (assert (pos? (count name)) "Can't find remote origin") + name (if (.endsWith name ".git") (subs name 0 (.indexOf name ".")) name)] [uri name]))) (defn dir @@ -332,7 +332,7 @@ (seq (map second plines)) (vec (reverse (first xs))) (vec (reverse (second xs))) - (->> lines + (->> lines (drop-while #(not= % "")) rest (interpose "\n") @@ -412,10 +412,10 @@ id)) parents)))]) tx (cond-> tx - (tempid? authorid) + (tempid? authorid) (conj [:db/add authorid :email/address author]) - - (and (not= committer author) (tempid? committerid)) + + (and (not= committer author) (tempid? committerid)) (conj [:db/add committerid :email/address committer]))] tx)) @@ -427,7 +427,7 @@ (mapv #(vector (subs % 0 40) (subs % 41 (count %))) - (line-seq s)))] + (line-seq s)))] commits)) (defn unimported-commits @@ -459,7 +459,7 @@ tx-ret @(d/transact conn [[:db/add temp :repo/uri repo-uri]]) repo (d/resolve-tempid (d/db conn) (:tempids tx-ret) temp)] (println "Adding repo" repo-uri) - repo))] + repo))] (doseq [commit commits] (let [db (d/db conn)] (println "Importing commit:" (:sha commit)) @@ -475,7 +475,7 @@ (doseq [a analyzers] (let [aname (az/keyname a) exts (az/extensions a) - srevs (set (map first (d/q '[:find ?rev :in $ ?a :where + srevs (set (map first (d/q '[:find ?rev :in $ ?a :where [?tx :tx/op :schema] [?tx :tx/analyzer ?a] [?tx :tx/analyzerRev ?rev]] @@ -484,7 +484,7 @@ ;;install schema(s) if not yet present (doseq [[rev aschema] (az/schemas a)] (when-not (srevs rev) - (d/transact conn + (d/transact conn (conj aschema {:db/id (d/tempid :db.part/tx) :tx/op :schema :tx/analyzer aname @@ -517,7 +517,7 @@ (catch Exception ex (println (.getMessage ex)) []))] - (d/transact conn + (d/transact conn (conj adata {:db/id (d/tempid :db.part/tx) :tx/op :analyze :tx/file f @@ -526,7 +526,7 @@ (println "Analysis complete!")) (defn main [& [db-uri commit]] - (if db-uri + (if db-uri (let [conn (ensure-db db-uri) [repo-uri repo-name] (get-repo-uri)] ;;(prn repo-uri) @@ -562,8 +562,8 @@ (d/q '[:find ?m :where [_ :code/text ?m] [(.contains ^String ?m "(ns ")]] db) (sort (d/q '[:find ?var ?def :where [?cn :code/name ?var] [?cq :clj/def ?cn] [?cq :codeq/code ?def]] db)) (sort (d/q '[:find ?var ?def :where [?cn :code/name ?var] [?cq :clj/ns ?cn] [?cq :codeq/code ?def]] db)) -(sort (d/q '[:find ?var ?def ?n :where - [?cn :code/name ?var] +(sort (d/q '[:find ?var ?def ?n :where + [?cn :code/name ?var] [?cq :clj/ns ?cn] [?cq :codeq/file ?f] [?n :node/object ?f] @@ -575,10 +575,10 @@ s (with-open [s (exec-stream (str \"git cat-file -p \" (:git/sha (d/entity db f))))] (slurp s)) adata (az/analyze a db s)] - (d/transact conn + (d/transact conn (conj adata {:db/id (d/tempid :db.part/tx) :tx/op :analyze :codeq/file f :tx/analyzer aname :tx/analyzerRev arev}))))") -) \ No newline at end of file +) diff --git a/src/datomic/codeq/util.clj b/src/datomic/codeq/util.clj index 76dcbe1..e2453bc 100644 --- a/src/datomic/codeq/util.clj +++ b/src/datomic/codeq/util.clj @@ -1,4 +1,4 @@ -;; Copyright (c) Metadata Partners, LLC. All rights reserved. +;; Copyright (c) Metadata Partners, LLC and Contributors. All rights reserved. ;; The use and distribution terms for this software are covered by the ;; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php) ;; which can be found in the file epl-v10.html at the root of this distribution. @@ -24,13 +24,4 @@ (or (index-get-id db attr x) (d/tempid :db.part/user))))) -(defmacro cond-> - [init & steps] - (assert (even? (count steps))) - (let [g (gensym) - pstep (fn [[pred step]] `(if ~pred (-> ~g ~step) ~g))] - `(let [~g ~init - ~@(interleave (repeat g) (map pstep (partition 2 steps)))] - ~g))) - (def tempid? map?)